Version-controlled relational dataset management

Abstract

The present invention relates to managing version-controlled datasets. An updated version-controlled dataset is created by applying a structured query to a specific version of a plurality of versions of the version-controlled dataset stored in a backing dataset. The data of the version including the specific version remains unchanged. A version-controlled dataset catalog is updated to include the structured query and lineage associated with the updated dataset version, the lineage indicating that the updated version-controlled dataset includes a successor version of the specific version. A dependency code associated with the updated version-controlled dataset is updated, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version. A database view of the updated version-controlled dataset is generated by selecting rows of the backing dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

Description

Background Art

[0001] The present invention, in some embodiments thereof, relates to data sets stored in a relational database, and more particularly to storing multiple versions of a data set in a relational database.

[0002] Relational databases store data in rows and columns. You can query a relational database to add new rows, add new columns, delete existing rows, delete existing columns, change values, perform intersection operations, and perform union operations. Summary of the Invention

[0003] According to a first aspect, an apparatus for managing a version-controlled dataset comprises: a processor and a non-transitory storage medium, the non-transitory storage medium carrying information, the information causing the processor to perform the following operations when executed by the processor: receiving at least one structured query, the at least one structured query indicating instructions for operating a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backup dataset; creating an updated version-controlled dataset in the backup dataset by applying the at least one structured query to the specific version; storing the updated version-controlled dataset in the backup dataset, wherein the data of the plurality of versions including the specific version remains unchanged; and updating a version-controlled dataset catalog to include at least one structured query and lineage associated with an updated version of a data set, the lineage indicating that the updated version-controlled data set includes successor versions of the specific version, wherein the version-controlled data set catalog includes historical structured queries and lineages applied in chronological order for each of the multiple versions; updating dependency codes associated with the updated version-controlled data set, the dependency codes indicating which data of the updated version-controlled data set matches the data of the specific version and which data of the updated version-controlled data set is deleted relative to the data of the specific version; generating a database view of the updated version-controlled data set by selecting rows of the backup data set corresponding to the updated version-controlled data set according to the dependency codes and projecting columns.

[0004] Versioned datasets store multiple versions of a dataset efficiently, using less memory and / or storage space to store the different versions. Each of the multiple versions is fully accessible and can be queried using structured queries.

[0005] According to a second aspect, a method for managing a version-controlled dataset comprises: receiving at least one structured query, wherein the at least one structured query indicates instructions for operating a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backup dataset; creating an updated version-controlled dataset in the backup dataset by applying the at least one structured query to the specific version; storing the updated version-controlled dataset in the backup dataset, wherein the data of the plurality of versions including the specific version remains unchanged; updating a version-controlled dataset catalog to include the at least one structured query and Lineage, the lineage indicating that the updated version-controlled dataset includes successor versions of the specific version, wherein the version-controlled dataset catalog includes historical structured queries and lineage applied in chronological order for each of the multiple versions; updating dependency codes associated with the updated version-controlled dataset, the dependency codes indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; generating a database view of the updated version-controlled dataset by selecting rows of the backup dataset corresponding to the updated version-controlled dataset according to the dependency codes and projecting columns.

[0006] According to a third aspect, a computer program product for a version-controlled dataset includes a non-transitory medium storing a computer program, wherein the computer program, when executed by at least one hardware processor, causes the at least one hardware processor to perform the following operations: receive at least one structured query, wherein the at least one structured query indicates instructions for operating a specific version of data, wherein the specific version is from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backup dataset; create an updated version-controlled dataset in the backup dataset by applying the at least one structured query to the specific version; store the updated version-controlled dataset in the backup dataset, wherein the data of the plurality of versions including the specific version remains unchanged; update the version-controlled dataset catalog to include The method comprises: comprising: providing at least one structured query and lineage associated with an updated version of a data set, the lineage indicating that the updated version-controlled data set includes successor versions of the specific version, wherein the version-controlled data set catalog includes historical structured queries and lineages applied in chronological order for each of the multiple versions; updating dependency codes associated with the updated version-controlled data set, the dependency codes indicating which data of the updated version-controlled data set matches the data of the specific version and which data of the updated version-controlled data set is deleted relative to the data of the specific version; and generating a database view of the updated version-controlled data set by selecting rows of the backup data set corresponding to the updated version-controlled data set according to the dependency codes and projecting columns.

[0007] According to a fourth aspect, an apparatus for managing a version-controlled dataset comprises: a processor for: receiving at least one structured query, the structured query indicating an instruction to operate a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backup dataset; creating an updated version-controlled dataset in the backup dataset by applying the at least one structured query to the specific version; storing the updated version-controlled dataset in the backup dataset, wherein the data of the plurality of versions including the specific version remains unchanged; updating a version-controlled dataset catalog to include the at least one structured query associated with the updated dataset version and the data along the backup dataset; The method comprises: updating a dependency code associated with the updated version-controlled dataset, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; and generating a database view of the updated version-controlled dataset by selecting rows of the backup dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

[0008] In another implementation of the first, second, third and fourth aspects, the backup dataset is implemented as a table comprising rows and columns, wherein the original version of the version-controlled dataset is stored in its entirety, and the subsequent versions of the version-controlled dataset are stored as incremental operations on the previous version by storing incremental rows and / or columns and by storing indications of operations on the previous version in metadata of the version-controlled dataset directory associated with the subsequent version.

[0009] The architecture of the backing dataset provides storage efficiency, for example, by using a single table and storing incremental changes to the original and / or previous versions rather than by storing each version separately.

[0010] In another implementation of the first, second, third and fourth aspects, the backing dataset comprises a single backing dataset storing the multiple versions of the version-controlled dataset.

[0011] In another implementation of the first, second, third and fourth aspects, information for compressing the single backup data set is further included, wherein the information includes a default value indicating deleted data.

[0012] In another implementation of the first, second, third and fourth aspects, the dependency encoding indicates which rows and / or columns of the updated version-controlled dataset match the rows and / or columns of the specific version, and which rows and / or columns of the updated version-controlled dataset are deletions relative to the specific version.

[0013] In another implementation of the first, second, third and fourth aspects, the dependency encoding is implemented as a bitmap, wherein, in response to new rows and / or columns being added to the updated version-controlled dataset, the dependency encoding bitmap includes a Boolean value for each corresponding row, and the Boolean value is set to TRUE if and only if the corresponding row participates in the updated version-controlled dataset, otherwise it is set to FALSE.

[0014] A bitmap is an efficient data structure (eg, in terms of storage requirements and / or processing requirements) for indicating which rows and / or columns belong to which version.

[0015] In another implementation of the first, second, third and fourth aspects, the backup data set stores a plurality of default values ​​indicating mappings to values ​​stored in a previous version.

[0016] Default values ​​(e.g., null values) can be compressed efficiently, reducing storage requirements compared to storing redundant values.

[0017] In another implementation of the first, second, third and fourth aspects, the information refers to code including additional extensions and / or an application programming interface (API), which connects to a relational database management system (RDBMS) that manages the version-controlled data set.

[0018] Traditional RDBMS systems cannot access historical data and / or do not store multiple versions of versioned datasets. Add-ons and / or APIs enhance existing RDBMSs by providing extended features for storing historical data and / or storing multiple versions of versioned datasets. RDBMSs are extended with code (e.g., dataset versioning extension code) with APIs, allowing users to collaboratively create, manipulate, and track versioned datasets.

[0019] In another implementation of the first, second, third and fourth aspects, metadata stored in the version-controlled dataset directory is shared by and readable by multiple users designated to access the version-controlled dataset, wherein concurrent operations for creating and / or revising the multiple versions of the version-controlled dataset requested by the multiple users on the version-controlled dataset are encapsulated as transactions.

[0020] Encapsulation allows multiple users to create and / or revise a versioned dataset atomically, where concurrent users cannot access or affect intermediate steps of the data flow of the versioned dataset.

[0021] In another implementation of the first, second, third and fourth aspects, the lineage of the corresponding versions stored in the version-controlled dataset directory represents the dependency relationship between the multiple versions of the version-controlled dataset and is selected from the group consisting of: the direct superior of the corresponding version, one superior among the multiple superiors of the previous version, the root superior indicating the first version without a previous version, and the direct superior of two or more merged versions.

[0022] In another implementation of the first, second, third and fourth aspects, the database view stores a mapping indicating calculations of columns and / or rows of the updated version-controlled dataset based on columns and / or rows of one or more previous versions, wherein the view definition of the updated version-controlled dataset provides instructions for applying the calculations to the columns and / or rows of the one or more previous versions based on the mapping.

[0023] Mappings can be used in the form of column and / or row calculations, for example, to avoid column name conflicts in backing datasets that may store data from multiple dataset versions that may have overlapping and conflicting column names. In some implementations, an internal canonical naming scheme for backing dataset columns is used (e.g., f0, f1, ...). For example, a mapping to the original user-provided column names, as defined by the initial structured query, can be provided within the view definition.

[0024] In another implementation of the first, second, third and fourth aspects, the calculation includes one or more members selected from the group consisting of: renaming, negation, type conversion, addition, cascading, union, any unary and n-ary mapping available in the RDBMS, and user-defined mapping.

[0025] Computations can be defined by the host RDBMS, thereby enabling compatibility with and / or allowing the use of existing RDBMSs.

[0026] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: re-execute the historical structured query applied in chronological order stored in the version-controlled dataset directory to create an updated version of the specific version of the multiple versions of the version-controlled dataset based on current real-time data; store the updated version in the version-controlled dataset; and update the version-controlled dataset directory using the lineage and instructions to re-execute the updated version.

[0027] A refresh operation can be used later to create a "newer" version of the original (or other lineage) versioned dataset based on the current state of the live data (or other updated data).

[0028] In another implementation of the first, second, third and fourth aspects, the at least one structured query includes instructions to insert new rows and / or new columns into the specific version, the backup dataset is updated to include the new rows and / or new columns, and the dependency encoding is updated to indicate that the new rows and / or new columns are associated with the updated version-controlled dataset, wherein the new rows and / or new columns are not associated with other versions.

[0029] Storage efficiency is provided by storing incremental new data of an updated versioned dataset rather than by storing the entire updated versioned dataset. Old versions are not affected and are maintained using metadata.

[0030] In another implementation of the first, second, third and fourth aspects, the view of the updated versioned dataset created by selecting rows of the backing dataset corresponding to the updated versioned dataset and projecting columns includes the new rows and / or new columns.

[0031] In another implementation of the first, second, third and fourth aspects, the at least one structured query includes instructions to delete rows and / or columns from the specific version, the version-controlled dataset catalog is updated to indicate that the rows and / or columns of the specific version are excluded from the updated version-controlled dataset, wherein the rows and / or columns are not deleted in the backup dataset, the dependency encoding is updated to indicate that the deleted rows and / or deleted columns are not associated with the updated version-controlled dataset, and the deleted rows and / or deleted columns remain associated with other versions.

[0032] When accessing a previous version of a dataset, the actual data for the deleted rows and / or columns remains in the backing table.

[0033] In another implementation of the first, second, third and fourth aspects, the view of the updated versioned dataset created by selecting rows of the backing dataset corresponding to the updated versioned dataset and projecting columns does not include the deleted rows and / or columns.

[0034] In another implementation of the first, second, third and fourth aspects, the at least one structured query includes instructions to: change values ​​in rows and / or columns of the particular version, copy the rows and / or columns affected by the changed values, update the backing dataset to include the copies, perform the change to the values ​​on the copies, and update the affiliation encoding to indicate that the changed copies with the values ​​are associated with the updated version-controlled dataset, wherein the changed copies with the values ​​are not associated with other versions.

[0035] In another implementation of the first, second, third and fourth aspects, unchanged values ​​in the copy associated with the updated version-controlled dataset are stored as default values, and the default values ​​indicate a mapping to corresponding data stored in the specific version, wherein the view of the updated version-controlled dataset is presented by replacing the default values ​​with the corresponding data according to the mapping.

[0036] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, enables the processor to detect orphan rows and / or columns in the backup dataset that are not associated with any version of the multiple versions, and automatically delete the orphan rows and / or columns from the backup dataset.

[0037] Removing orphan rows and / or columns improves storage efficiency by removing data that is not used by any version of the versioned dataset.

[0038] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: analyze configuration parameters, which indicate the trade-off between storage cost and access cost of one or more of the multiple versions and the updated version-controlled data set; create a new backup data set for one or more versions, wherein the configuration parameters indicate that the storage cost is lower than the access cost.

[0039] Nesting of previous versions may result in decreased scan performance and / or increased cost to extract the separate versions of the data from the backing dataset. The separate versions are stored in a new backing dataset to improve access and / or storage performance.

[0040] In another implementation of the first, second, third and fourth aspects, information is further included, which, when executed by the processor, causes the processor to perform the following operations: convert the at least one structured query into an abstract syntax tree (AST); operate the AST by replacing attribute references to obtain an operated AST; transform the operated AST into a rewritten structured query; and apply the rewritten structured query to the backup data set.

[0041] The conversion, manipulation, and transformation of structured queries enable users to transparently use standard queries such as SQL to operate on specific versions stored in the backing dataset, thereby achieving compatibility with the RDBMS that manages the backing dataset.

[0042] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: prepare the view and / or the backup dataset using rewrite rules for processing at least one structured query to operate the updated version-controlled dataset, and apply the at least one structured query to the current version of the view and / or the backup dataset to operate the updated version-controlled dataset.

[0043] The transformation uses implicit rewriting, without having to manipulate the AST to obtain the manipulated AST, or transforming the manipulated AST into a rewritten structured query. The user query is applied directly to the view, and the backing tables and version-controlled dataset catalog are updated according to the rewrite rules.

[0044] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: select one of the multiple versions stored in the backup dataset; verify whether the selected version has no dependent versions; apply at least one structured query to the selected version; update the metadata of the version-controlled dataset catalog associated with the selected version to further include at least one structured query applied to the existing historical structured query applied in chronological order; and update the view of the selected version based on the at least one structured query applied.

[0045] An existing dataset can be modified instead of creating a new version.

[0046] In another implementation of the first, second, third and fourth aspects, wherein: the at least one structured query includes instructions to sample and / or partition at least one subset of the specific version; creating includes creating the updated version-controlled dataset including the at least one subset; wherein the version-controlled catalog of the updated version-controlled dataset includes a lineage, and the lineage indicates that the at least one subset of the updated version-controlled dataset is derived from the specific version.

[0047] For example, the sampled dataset can be used to create different datasets for training and / or validating ML models.

[0048] In another implementation of the first, second, third and fourth aspects, wherein: the at least one structured query includes an instruction to randomly permute the data of the specific version; creation includes creating the updated version-controlled dataset including the randomly permuted data, wherein the randomly permuted data is implemented as at least one of: a mapping of the data of the specific version to a new order of the updated version-controlled dataset, and a replication of the data of the specific version to the new order of the updated version-controlled dataset; the version-controlled directory of the updated version-controlled dataset includes an instruction indicating that the updated version-controlled dataset with the randomly permuted data is a lineage derived from the specific version, and indicates random permute of the data of the specific version to create the updated version-controlled dataset; and a dependency mapping stores the new order of the updated version-controlled dataset.

[0049] For example, randomly permuting a dataset can be used to create different datasets for training and / or validating ML models.

[0050] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: receive an instruction to clear the specific version; delete data from the backup dataset that is exclusive to the specific version and does not exist in any dependent dataset derived from the specific version; delete metadata from the version-controlled dataset directory that is exclusive to the specific version and does not exist in any dependent dataset derived from the specific version; and update the metadata of the version-controlled dataset directory for any dependent dataset derived from the specific version by changing the dependency from the cleared version to an earlier version on which the cleared version depends.

[0051] The purge operation may attempt to free up occupied storage capacity of the purged dataset while keeping the integrity (eg, data and history) of possible dependent version-controlled datasets intact and unaffected.

[0052] In another implementation of the first, second, third and fourth aspects, information is also included, which, when executed by the processor, causes the processor to perform the following operations: receive an instruction to merge a first version with a second version, wherein the first version and the second version are derived from a common superior version; create a merged set of operations by integrating operations applied to the first version with operations applied to the second version; apply the merged set of operations to the common superior version; and store an indication of the lineage derived from the first version and the second version and the merged set of operations in the metadata of the version-controlled dataset catalog associated with the updated version-controlled dataset.

[0053] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as those known to those of ordinary skill in the art. Although methods and materials similar or equivalent to those described herein can be used to test or perform embodiments of the present invention, exemplary methods and / or materials are described below. In the event of a conflict, this patent specification (including definitions) shall prevail. In addition, these materials, methods, and examples are illustrative only and are not necessarily limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Some embodiments of the present invention have been described herein by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings, it should be emphasized that the details shown are for illustration only and for purposes of illustrative discussion of the embodiments of the present invention. In this regard, it will be apparent to those skilled in the art from the accompanying drawings how to implement the embodiments of the present invention.

[0055] In the attached figure:

[0056] Figure 1 A flowchart of a method of managing a version-controlled dataset according to some embodiments;

[0057] Figure 2 is a block diagram of components of a system for managing version-controlled datasets of some embodiments;

[0058] Figure 3 A flowchart depicting exemplary operations performed on version-controlled data according to some embodiments;

[0059] Figure 4 A diagram for visually depicting exemplary operations performed on version-controlled data according to some embodiments;

[0060] Figure 5 An exemplary state diagram depicting changes between exemplary published, archived, prepared, and cleared states of some embodiments;

[0061] Figure 6 A block diagram of an exemplary implementation of code appendage components for a transactional relational database management system (RDBMS) of some embodiments, the RDBMS being used to manage multiple versions of a data set and the flow of data between these components;

[0062] Figures 7A-7B A data flow diagram of an exemplary process for rewriting a structured query to be implemented on a specific version of a backing dataset of some embodiments;

[0063] Figure 8 including a data flow diagram depicting conflicting and non-conflicting operations of some embodiments;

[0064] Figure 9 A diagram illustrating an exemplary layout of a backing data set storing multiple versions of a data set according to some embodiments;

[0065] Figure 10 A diagram illustrating an exemplary storage implementation for storing multiple versions of a version-controlled dataset in accordance with some embodiments;

[0066] Figure 11 A diagram depicting an exemplary machine learning (ML) workflow utilizing multiple versions of a version-controlled dataset in accordance with some embodiments. DETAILED DESCRIPTION

[0067] The present invention, in some embodiments thereof, relates to data sets stored in a relational database, and more particularly, but not exclusively, to storing multiple versions of a data set in a relational database.

[0068] At least some implementations of systems, methods, devices, processors, and / or code instructions (also referred to as information) manage a backing dataset that optionally includes multiple versions of a versioned dataset within a single dataset (e.g., a table). A specific version of the multiple versions can be operated on by one or more structured queries. The structured queries include instructions for operating on structured data stored in a structured dataset, i.e., the backing dataset, such as inserting a row, deleting a row, adding two columns together and providing the output to a third column, and changing the value in a cell. An updated versioned dataset is created in the backing dataset by applying one or more structured queries to a specific version. The specific version is a parent version, and a new version created by applying the one or more structured queries to the specific version is a successor to the parent version. The updated versioned dataset is stored in the backing dataset. The data from the multiple previously stored versions remains unchanged. The updated versioned dataset stores incremental changes to the specific version (created by applying the one or more structured queries to the specific version) without generating redundant copies of the specific version's data. For example, when a structured query is an instruction to add another row to a specific version, the additional row is added to the updated version without copying the remaining data of the specific version. The data of the previous version remains unchanged. The versioned dataset catalog includes the historical structured queries applied in chronological order and the lineage of each version of the backing dataset. The versioned dataset catalog is updated to include one or more structured queries applied to the specific version, as well as a lineage indicating that the updated versioned dataset is a successor of the specific version. The dependency encoding is updated to indicate which data of the updated versioned dataset matches the data of the specific version and which data of the updated versioned dataset is deleted relative to the data of the specific version. For example, when the specific version includes 10 rows of data and 1 row is deleted according to the structured query, the dependency encoding is updated to indicate that the updated versioned dataset includes 9 rows of data of the specific version, excluding the deleted row. It should be noted that the dependency encoding of the specific version indicates that 10 rows exist, and the updated versioned dataset stores the data of the 10 rows of the specific version because the data of the specific version (and previous versions) are not affected. The dependency encoding can store the order of the data of the specific version. For example, if the 10 rows of a particular version are shuffled into a new order, the dependency encoding stores the new order. The versioned dataset retains the original 10 rows of the particular version rather than recopying them into the new order, saving space because the order is stored in the dependency encoding.

[0069] It should be noted that some embodiments refer to relational datasets with schemas, but this is not necessarily limiting, as at least some embodiments are applicable to other implementations, such as schema-less data, documents, graph data, etc., when units of work can be expressed as compact descriptive abstractions, such as query languages ​​and basic infrastructure for managing dataset versions is available.

[0070] At least some implementations of the systems, methods, apparatus, and / or code instructions described herein relate to collaborative and automated operations by collecting and / or sharing metadata of annotated, generally immutable, version-controlled datasets and / or by using shared backing tables and database views for efficient storage.

[0071] At least some implementations of the systems, methods, apparatus, and / or code instructions described herein address the technical problem of storing different versions of a dataset, e.g., in the field of machine learning, on which queries may be performed, e.g., for training, validating, and / or inferencing ML models.

[0072] At least some implementations of the systems, methods, apparatus, and / or code instructions described herein improve the technical field of databases and / or machine learning by allowing storage of different versions of a dataset against which queries may be performed, e.g., for training and / or validating ML models.

[0073] Traditionally, the primary purpose of a database is to reflect the latest state of live data (e.g., production data serving business logic) that is stored and continuously updated in the system. However, there are use cases where it is desirable to selectively retain historical subsets of data in a way that allows queries to be rerun against that data, thereby providing stable, consistent, and reproducible query results based on the state of the data at a certain point in time.

[0074] Examples of usage patterns include: deep branches covering (a) a history of versions, where each successor version depends on the previous version; (b) flat branches with alternative versions that all depend on a common previous version; (c) a "scientific pattern" using separate, independent branches from an evolving mainline, where these branches are used for reproducible experiments but are never merged back, and the mainline is a series of successor versions, each of which depends on the previous version, and the previous version may be associated with multiple alternative versions; (d) a typical "data governance pattern" found in machine learning environments, where data scientists perform collaborative data cleanup simultaneously based on different versions and eventually merge changes back to the mainline, for example, complex branches obtained by combining previous and subsequent versions, and multiple alternative successor versions.

[0075] Traditional RDBMSs support the above features (e.g., (a), (b), (c), and (d)) only to a very limited extent by providing a snapshot isolation mechanism for query processing, which provides concurrency control and isolates transactional queries from ongoing modifications in concurrent transactions. Snapshot isolation is a core feature of RDBMSs that allows queries to be run repeatedly and retrieve consistent results within the context of a transaction.

[0076] While snapshot isolation is sufficient for processing queries, it does not satisfy any of the scenarios described in this article. These snapshots are volatile because they are limited to the duration of the ongoing transaction. When that transaction ends and a new transaction begins, a new volatile snapshot is created, reflecting the more recent, consistent state of the database. When the transaction that created the snapshot terminates, the historical snapshot at that point in time is irrecoverably lost. Ultimately, all transactions participate in a single linear history (often called a serial transaction plan), and branching is not supported. Furthermore, these snapshots can have a significant negative impact on the concurrency, resource usage, and performance of the entire system, so keeping transactions open for extended periods of time is not tolerated.

[0077] Existing database designs do not address or only partially address and / or do not adequately address the technical issues described herein, such as:

[0078] (Relational) database systems only provide information about the most recent state of data (at the time of the transaction); historical data is overwritten and irreversibly lost. Therefore, snapshot isolation only provides a solution for very short durations (single transactions). Snapshot isolation for long read transactions incurs significant overhead, especially in the presence of concurrent write transactions. Branching is not supported at all, as transactions are serialized into a single, continuous plan.

[0079] Temporal databases support only a single common history, with concurrent modifications serialized into a single, continuous plan based on their transaction times. Fine-grained versions are automatically created for each committed transaction. Tracking all changes at this granular level would incur significant overhead in storing and processing both historical and real-time data, and in practice, this fine-grained versioning might never be necessary.

[0080] Custom machine learning solutions have limited integration with database systems that store raw data. Due to redundancy between versions of a dataset, storage overhead is high, schema changes are generally not supported, and they rely on external storage (files) and the corresponding transformation overhead. The system integration and performance (query capabilities, indexing, type compatibility) provided are limited, and there are data security and privacy vulnerabilities.

[0081] More specifically, common relational database systems do not support data scientists collaborating on version-controlled datasets, a typical practice in the ML field. This makes the typical exploratory, trial-and-error workflow of model training cumbersome and expensive. Data scientists are required to manually provide and maintain the provenance of trained ML models on training datasets, which is crucial for reproducible model training. Complex and repetitive data governance processes discourage sharing and reuse, and lack automation.

[0082] At least some implementations of the systems, methods, devices, and / or code instructions described herein provide improvements over other approaches that attempt to solve the technical problems described herein. For example:

[0083] Some temporal databases attempt to partially address the technical issues described in this article, but they are still not suitable for many use cases. These approaches add the classic RDBMS concept of "transaction time", which determines the validity of a volatile snapshot, to the concept of "valid time", allowing queries to the database at a certain point in time, thus allowing access to the historical state of the data.

[0084] Temporal databases do not support collaboration and / or concurrent work with different versions of a data set (implemented according to at least some embodiments described herein) because these temporal databases only support a single linear history of events, making branching unsupported, whereas more comprehensive data version control systems (implemented according to at least some embodiments described herein) intended to cover a wider range of use cases are designed to support graph-like structures of alternative histories, much like the concept of branches in source control management systems (e.g., git).

[0085] Schema evolution techniques track schema changes for relationships. However, a dataset only has a single state along multiple historical schema transformations, which is insufficient for storing the dataset's history. Even combining temporal database extensions with schema evolution still misses the potential for supporting nonlinear histories. Furthermore, due to version compatibility issues, schema evolution techniques have limited support for Data Definition Language (DDL) operations and generally do not support Data Manipulation Language (DML) operations.

[0086] Version control software (e.g., git, svm, etc.) can capture some concepts of versioning and branching for datasets represented in text formats, such as comma-separated value (CSV) files. However, CSV files have significant disadvantages compared to the solutions required for database environments: (a) CSV files lack full query and indexing capabilities (SELECT); (b) CSV files lack a schema and have no concept of attributes, so version control software cannot selectively update attribute values ​​using update statements, nor can it perform operations such as adding or removing attributes from CSV files; (c) version control software is a text-based approach that applies patches, so users cannot register operations in the history (e.g., delete rows if a condition is met); (d) splitting datasets based on predicates is not feasible; (e) version control software creates a copy of each version accessed by each user at a given time, which is unreasonable for shared databases with concurrent users accessing multiple versions; and (f) CSV files exist independently of the database system and require separate precautions for data type conversion, recovery, and privacy.

[0087] DVC is an open source project that introduced the concept of pipelines as part of its commit message. A pipeline is a series of Python scripts that transform data. This allows users to execute arbitrary code after obtaining the data, but does not address previous limitations.

[0088] Dedicated solutions exist for machine learning, but these approaches often lack deep integration with the full RDBMS stack or suffer from limitations, such as reliance on external, file-based storage or mismatches in type mapping and inference, and a lack of support for schema changes between versions of a dataset.

[0089] At least some implementations of the systems, methods, apparatus, and / or code instructions described herein provide one or more of the following potential technical advantages and / or potential technical effects:

[0090] ●In addition to processing real-time data (e.g., production data serving business logic, ML training datasets), it also supports relational dataset versioning.

[0091] ● A complete workflow for creating version-controlled datasets from previously unversion-controlled data.

[0092] ●Evolutionary development (applying data and schema modifications) of new versions of datasets based on unversioned datasets and versioned datasets.

[0093] ● Efficient management of version-controlled relational data instances, especially minimizing redundancy and compact storage.

[0094] ● Efficiently and completely support full SQL query capabilities for all versions of available data sets, while ensuring consistent and repeatable query results for historical data.

[0095] ●Preserve the complete lineage of version-controlled data, that is, preserve not only the data itself, but also the history of the operations that created and modified the data.

[0096] ●Supports fully concurrent operations, especially lock-free reading of version-controlled data.

[0097] ●Encourage collaboration and reuse when developing evolving versions of datasets.

[0098] ● Provide deep integration with core RDBMS features, particularly data privacy, permissions, and recovery.

[0099] At least some implementations of the systems, methods, apparatus, and / or code instructions described herein provide one or more of the following features:

[0100] ●Any DML operation, such as INSERT / UPDATE / DELETE on version-controlled data.

[0101] ●Any DDL operation, such as: alter table...add, drop, rename, set.

[0102] ● Efficient storage, avoiding the redundancy common among related versions of a dataset.

[0103] ● Denormalized storage, applying compression but avoiding potentially complex and expensive operations, such as joins.

[0104] ● Extraction primitives with low complexity and high optimization potential (e.g., selection / projection pushdown).

[0105] ●MPP distributed storage and access.

[0106] ●Index on versioned datasets, e.g., partial indexes with dependency encoding predicates.

[0107] ● Additional storage optimization using column encoding and function attribute mapping to improve compression.

[0108] ●Storage reorganization (reordering) by user attributes to improve compression.

[0109] ●Storage reorganization (reordering) through dependency encoding to reduce version fragmentation in the public lookaside table.

[0110] ● Transparently re-encode the storage representation of each dataset version internally between a single lookaside table and a shared lookaside table at any time, controlling the trade-off between space and performance.

[0111] ●Fully integrated into the host RDBMS, including interoperability of version-controlled and live data, permissions, ACID, recovery, ingestion tools, backups, etc.

[0112] ● As an embodiment, full ML dataset lineage is supported (not limited to ML).

[0113] ●Sources of datasets that support trained ML models.

[0114] ●Support exploration, experimentation, and verification during ML model training.

[0115] ●Support collaboration and reuse of automation in repetitive and expensive dataset preparation processes.

[0116] ●Supports seamless refreshing or incremental supplementation of existing management and control historical datasets with the latest data from real-time datasets.

[0117] ●Fully integrated into live data, direct query, and data transfer on both versioned and unversioned datasets.

[0118] ●Data integrity is fully automated and there is no type conversion mismatch.

[0119] ●A public permissions system helps extend existing data privacy policies to version-controlled datasets.

[0120] ●Full RDBMS integration (transactions, recovery, backup, ...).

[0121] The Dataset Versioning extension can be added to an existing production RDBMS at any time to equip the system with dataset versioning capabilities. The extension installs executable functionality (i.e., code) for managing versioned datasets and creates an empty repository to store the versioned datasets.

[0122] A lookaside table is a versioned data repository that is not necessarily accessible to users. It consists of user columns with the same type definitions as the original data, with mapped column names (to avoid naming conflicts) and optionally with auxiliary columns for version management (dependency encoding) to compactly store several versions of a dataset within a common lookaside table. Shared lookaside tables improve the storage efficiency of versioned datasets by avoiding redundancy.

[0123] ● Denormalized storage in lookaside tables with no joins for data extraction (sparseness).

[0124] ● Compression addresses the sparsity of potentially wide lookaside tables.

[0125] ●Dataset version views expose versioned data to users (read-only, authorizable, managed projection list, encoding).

[0126] ●Operations: insert, delete, update (row or column by row) by rewriting, for example, through an updatable view.

[0127] ●Schema changes that are version controlled: add (add), remove (hide), change column name (rename) / type (convert or copy) / collate (copy) / constraint (copy).

[0128] ●If there is no dependent dataset, deleting a single dataset is supported.

[0129] ●Supports integration and compression of outdated intermediate dataset versions while retaining dependent version-controlled datasets intact.

[0130] ●Configurable trade-off between storage utilization and performance (dataset creation, data access).

[0131] Before explaining at least one embodiment of the present invention in detail, it should be understood that the application of the present invention is not necessarily limited to the details of construction and arrangement of components and / or methods set forth in the following description and / or shown in the drawings and / or examples. The present invention is capable of other embodiments or can be implemented or carried out in various ways.

[0132] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions, wherein the computer-readable program instructions cause a processor to perform various aspects of the present invention.

[0133] The computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices.

[0134] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network.

[0135] The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), and may also be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit comprising, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instructions by personalizing the electronic circuit using state information of the computer-readable program instructions to perform various aspects of the present invention.

[0136] Various aspects of the present invention are described herein in conjunction with flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0137] The flowcharts and block diagrams in the figures illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products of various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, segment or partial instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions described in the box may not be implemented in the order described in the figure. For example, in fact, the two boxes shown in succession can be executed almost simultaneously, or sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented by a system based on special-purpose hardware, which performs specific functions or actions, or performs a combination of special-purpose hardware and computer instructions.

[0138] Now refer to Figure 1 , which is a flowchart of a method of managing a version-controlled dataset in some embodiments. Figure 2, which is a block diagram of components of a system 200 for managing version-controlled datasets in some embodiments. The system 200 may be implemented by executing code instructions (e.g., code 206A) stored in a memory 206 by a processor 202 of a computing device 204. Figure 1 and the actions of the methods described in the other figures described herein. Figure 3 , which is a flowchart depicting exemplary operations performed on version-controlled data in accordance with some embodiments. Figure 4 , which is a diagram that intuitively depicts exemplary operations performed on version-controlled data in some embodiments. Figure 5 , which is an exemplary state diagram depicting changes between exemplary Published 502, Archived 504, Prepared 506, and Cleared 508 states of some embodiments. The Published state 502 is also Figure 4 The archive state 504 is still Figure 4 414 is shown intuitively. Now refer to Figure 6 , which is a block diagram of an exemplary implementation of code-attached components of a transactional relational database management system (RDBMS) of some embodiments for managing multiple versions of a data set and the flow of data between these components. Figures 7A-7B , which is a data flow diagram of an exemplary process for rewriting a structured query to be implemented on a specific version of a backing data set in some embodiments. Figure 8 , which includes a data flow diagram depicting conflicting and non-conflicting operations of some embodiments. Figure 9 , which is a diagram depicting an exemplary layout of a backing data set storing multiple versions of a data set in some embodiments. Figure 10 , which is a diagram illustrating an exemplary storage implementation for storing multiple versions of a version-controlled data set in some embodiments. Figure 11 , which is a diagram depicting an exemplary ML workflow of some embodiments utilizing multiple versions of a version-controlled dataset.

[0139] Now refer back to Figure 9, lookaside table 902 stores multiple versions of the versioned dataset, denoted as V0 904, V1 906, V2 908, and V3 910. Tree 912 depicts the lineage of the versioned dataset, where V0 904 is the initial version, versions V1 906 and V3 910 are alternatives that are both derived from V0 904, and V2 908 is derived from V1 906. As shown in lookaside table 902, V0 904 stores a complete set of rows and columns. For each of V1 906, V2 908, and V3 910, the data changes on the corresponding parent version are stored in table 902, that is, for V1 906 and V2 908, the changes on V0 904 are stored, and for V2 908, the changes on V1 906 are stored. The black area 918 represents a default value, such as NULL, which may represent a portion of the parent version that is excluded from the specific version, such as rows, columns, and / or data that are deleted in the specific version but appear in the parent version.

[0140] A dependency code 912 is provided for each version, i.e., V0 904, V1 906, V2 908, indicating the columns of the lookaside table 902 (e.g., one or more parent versions) included in the particular version. A dependency code 914 is provided for each version, i.e., V0 904, V1 906, V2 908, indicating the rows of the lookaside table 902 (e.g., one or more parent versions) included in the particular version.

[0141] Equation 916 is an exemplary algebraic representation for generating each of V0, V1, V2, and V3 from lookaside table 902 by applying dependency encodings 912 and 914 to lookaside table 902.

[0142] Figure 9 Four versions of a single dataset are depicted, interrelated using deep nesting and branching, as shown in a tree 912. In the exemplary embodiment, user data is organized in a lookaside table T0 902 in rows and columns. The user data representing the actual versioned dataset is represented by columns corresponding to the original definition of the data from the live dataset, shown as 904. The lookaside table 902 is organized with auxiliary columns (v) representing dependency codes 912. n ) extension (ie, in some embodiments, the dependency bitmap of each dataset version). The projection vector p of each dataset version n 912 Metadata stored for each dataset version (e.g., in a dataset view definition). n (as in the dependency code 912) and the corresponding selection v n Applied to the complete data of T0 (as in dependency encoding 914), the data of the separate dataset versions can be extracted cheaply.

[0143] The corresponding storage layout represents a wide (and growing as new columns are added for new dataset versions) table T0, where new rows for new versions are appended. Due to the relational but "denormalized" table storage, this layout may resemble a sparse matrix of entries, where, for example, large areas of consecutive rows may be set to NULL when they do not participate in any dataset version.

[0144] To efficiently support this storage model, the host RDBMS provides table storage, which allows adding columns to an existing table T0. For example, columns can be added cheaply in terms of low data storage and / or reduced processor utilization. Tables storing potentially large portions of matrices with NULL values ​​can be appropriately compressed. It should be noted that cheap column addition and data compression are not absolutely necessary. The corresponding costs can be used in the cost model for automatic storage selection, as described in this article.

[0145] Now refer back to Figure 2 , multiple architectures of the system 200 based on the computing device 204 can be implemented. In an exemplary implementation, the computing device 204 storing the code 206A can be implemented as one or more servers (e.g., network servers, web servers, computing clouds, virtual servers) that provide centralized services for storing and / or operating data sets and / or running queries on data sets to one or more client terminals 212 via a network 214 (e.g., reference Figure 1 and / or one or more actions described in 3 and / or other figures), for example, providing software as a service (SaaS) to the client terminal 212, providing software services that can be accessed using the software interface 206B (e.g., application programming interface (API), software development kit (SDK)), providing applications for local download to the client terminal 212, and / or providing functionality to the client terminal 212 using a remote access session, such as through a World Wide Web browser. For example, multiple users use their respective client terminals 212 to provide one or more structured queries 212A for creating a new version of the version-controlled dataset 216B and / or for accessing different versions of the version-controlled dataset 216B stored on the centralized computing device 204. In another example, the computing device 204 may include executing a reference Figure 13 and / or other figures, for example, as a standalone client terminal designed to be used by a user of the client terminal. In this implementation, different versions of the version-controlled dataset 216B are stored locally and / or implemented by the corresponding client terminal, and the user uses the corresponding client terminal to enter a structured query 212A to be run on the version-controlled dataset 216B. Alternatively or in addition, the implementation reference Figure 1 The code 206A that implements the features of the method described in

[0065] and / or

[0066] 3 can be implemented as an addition to an existing RDBMS 250 system, and / or the code 206A can be connected to the existing RDBMS 250 via a software interface such as an API 206B. In implementations where the code 206A is an addition and / or extension to the RDBMS 250, the code 206A can sometimes be referred to as a versioned dataset extension. It should be noted that the term versioned dataset extension provides an exemplary implementation of the code, but the term versioned dataset extension can sometimes be substituted with code that is not necessarily an addition and / or extension.

[0146] The processor 202 of the computing device 204 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), and an application specific integrated circuit (ASIC). The processor 202 may include a single processor, or multiple processors (homogeneous or heterogeneous) arranged for parallel processing, as a cluster, and / or as one or more multi-core processing devices.

[0147] The data storage device 206 stores code instructions that can be executed by the processor 202, such as random access memory (RAM), read-only memory (ROM), and / or storage devices such as non-volatile memory, magnetic media, semiconductor memory devices, hard disk drives, removable storage, and optical media (e.g., DVD, CD-ROM). The storage device 206 stores code 206A, which, when executed by the processor 202, implements the reference Figure 1 The storage device 206 may store code for interfacing with the RDBMS 250 and / or the client terminal 212, such as the API 206B.

[0148] The computing device 204 may include a data repository 216 for storing data, such as a backing dataset repository 216A that stores one or more backing datasets, each of which stores a corresponding versioned dataset 216B, a versioned dataset catalog 216C that may store metadata records for each version of the versioned dataset 216B, and a dependency encoding 216D. The data repository 216 may be implemented as, for example, a memory, a local hard drive, a virtual memory, a removable storage unit, an optical disk, a storage device, and / or a remote server and / or a computing cloud (e.g., accessed using a network connection).

[0149] The network 214 may be implemented as, for example, the Internet, a local area network, a virtual private network, a wireless network, a cellular network, a local bus, a point-to-point link (eg, wired), and / or combinations thereof.

[0150] The computing device 204 may include a network interface 218 for connecting to the network 214, such as one or more of a network interface card, a wireless interface for connecting to a wireless network, a physical interface for connecting to a cable for a network connection, a virtual interface implemented in software, network communication software providing a higher-layer network connection, and / or other implementations.

[0151] Computing device 204 may be connected to server 210 and / or client terminal 212 using network 214 (or another communication channel, e.g., via a direct link (e.g., cable, wireless) and / or an indirect link (e.g., through an intermediate computing unit such as a server and / or through a storage device). Server 210 may, for example, store updated versions of RDBMS 250 and / or code 206A. Client terminal 212 may be used by a user who remotely accesses computing device 204, as described herein.

[0152] The computing device 204 and / or the client terminal 212 includes and / or communicates with one or more physical user interfaces 208, which include mechanisms for a user to optionally enter data and / or view data within a GUI (e.g., enter a query for a selected version of a dataset and / or view a selected version of a dataset). Exemplary user interfaces 208 include, for example, one or more of a touch screen, a display, a keyboard, a mouse, and voice-activated software using a speaker and a microphone.

[0153] Optionally, at 102, code including an add-on extension and / or application programming interface (API) for connecting to a relational database management system (RDBMS) that manages version-controlled datasets is provided and / or installed and / or accessed. Traditional RDBMS systems cannot access historical data and / or do not store multiple versions of version-controlled datasets. The add-on and / or API enhances existing RDBMSs by providing extended features for storing historical data and / or storing multiple versions of version-controlled datasets. The RDBMS is extended with code (e.g., dataset versioning extension code) with the API to allow users to collaboratively create, manipulate, and track version-controlled datasets.

[0154] Alternatively or in addition, the code includes the fully integrated functionality described herein for managing version-controlled datasets.

[0155] The code can be accessed by one or more users. Multiple users can create different versions of the version-controlled dataset, which can be merged together and / or processed in other ways as described herein.

[0156] As used herein, a user is an exemplary use case and is not necessarily limited to a human user. A user may be an automated user (e.g., a robot, code), and / or may refer to other externally and / or internally provided instructions from any source.

[0157] For example, users are using an RDBMS to create training datasets to train ML models and / or business cases. The RDBMS can manage real-time data, such as tables and / or views, which may be subject to access permissions defined and / or enforced within the RDBMS. Real-time data may be updated concurrently and / or inaccessible to historical data.

[0158] An RDBMS can be extended using code (e.g., a dataset versioning extension) and / or using an API to allow users to collaboratively create, manipulate, and / or track versioned datasets. An RDBMS can be extended with a dataset versioning repository designed to store multiple versions of managed datasets. The repository may initially be empty, i.e., have no versioned data. The repository may be part of an installation. A versioned dataset catalog may initially include empty tables for storing metadata (e.g., history, lineage, documentation, etc.) on the versioned datasets. The metadata may be shared and / or readable by all users participating in dataset versioning, optionally represented by a user group / role in the RDBMS. This user group may also have permission to use the dataset versioning extension API.

[0159] As described herein, any user with read access to a given existing versioned dataset can create a dependent version through the Dataset Versioning Extension API. A unique name for the parent versioned dataset, a unique name for the new versioned dataset, and / or a list of data manipulation statements (INSERT / UPDATE / DELETE / ALTER) can be provided. Optionally, the user provides a text comment describing the rationale / purpose of the revised dataset version. A new versioned dataset can be conceptually created using the data from the parent version and the operations applied. The parent versioned dataset remains unchanged.

[0160] At 104, one or more structured queries are optionally received from a user. The structured queries may be in (and / or transformed into) structured query language (SQL). The structured queries indicate instructions for operating on data of a specific version among multiple versions included in a version-controlled dataset. The version-controlled dataset is stored in a backup dataset. The specific version may be a terminal node in a graph (or tree, or linear chain) of dependent versions derived from a common source version. In some implementations, the specific version may be an intermediate node in a graph (or tree, or linear chain) of dependent versions, for example, for clearing and / or creating a new branch.

[0161] SQL is a domain-specific language for programming, designed for managing data stored in an RDBMS or for stream processing in an RDBMS. Structured queries may meet one or more of the following characteristics:

[0162] ● Set-oriented: used to access many records through a single command.

[0163] ● Declarative: No need to specify how to reach the record, such as with an index or without an index.

[0164] Equivalence with relational algebra: SQL, originally based on relational algebra and tuple-relational calculus, consists of multiple types of statements that can be informally categorized as sublanguages, typically: data query language (DQL), data definition language (DDL), data control language (DCL), and data manipulation language (DML). The scope of structured query (e.g., in SQL) includes data query, data manipulation (insert, update, and delete), data definition (schema creation and modification), and data access control. While SQL is essentially a declarative language (4GL), it also includes procedural elements.

[0165] Optionally, a structured query is received from multiple users who specify access rights to a versioned dataset. Concurrent operations requested by multiple users on the versioned dataset to create and / or revise multiple versions of the versioned dataset are encapsulated as transactions. This encapsulation allows multiple users to atomically create and / or revise the versioned dataset, without the concurrent users being able to access or influence intermediate steps in the data flow of the versioned dataset. Metadata stored in the versioned dataset catalog is shared by and readable by the multiple users.

[0166] As used herein, the terms structured query, operation, and transaction are sometimes used interchangeably. The terms structured query, operation, and transaction may refer to data operations performed on the data sets described herein. The term SQL is exemplary and is not necessarily limited to standard SQL and may refer to other methods for expressing instructions for operating on data sets.

[0167] A user can create a first (i.e., initial) version of a dataset (e.g., via the Dataset Versioning Extension API), and / or can create subsequent versions of an existing dataset. The user can provide a unique name for the new versioned dataset, and / or can provide an SQL query that defines the first and / or subsequent versions of the data. The SQL query can select arbitrary data accessible to the user from a "live data" domain (e.g., in its latest, volatile state) and / or from an existing versioned dataset (e.g., in an immutable, stable state). Optionally, the user can provide a list of data manipulation statements in SQL (e.g., INSERT / UPDATE / DELETE / ALTER) to be applied to the initial version of the dataset. Optionally, the user provides a text comment describing the rationale / purpose for creating the dataset version.

[0168] refer to Figure 3Additional exemplary structured queries are described.

[0169] Optionally, one or more structured queries are converted and / or processed to operate on the backing table at 106. The conversion, manipulation, and transformation of the structured queries enable users to transparently use standard queries such as SQL to operate on a version stored in the backing dataset, thereby achieving compatibility with the RDBMS that manages the backing dataset.

[0170] In one exemplary implementation, a structured query is converted into an abstract syntax tree (AST). The AST is manipulated by replacing attribute references to obtain a manipulated AST. The manipulated AST is converted into a rewritten structured query. The rewritten structured query is applied to a backup dataset.

[0171] In another exemplary implementation, a view and / or lookaside table are prepared using rewrite rules for processing at least one structured query to operate on an updated versioned dataset. The at least one structured query is applied to a current version of the view and / or lookaside table to operate on the updated versioned dataset. The transformation uses implicit rewriting, without manipulating the AST to obtain the manipulated AST or transforming the manipulated AST into a rewritten structured query. The user query is applied directly to the view, and the lookaside table and the versioned dataset directory are updated according to the rewrite rules.

[0172] Now refer back to Figure 7A , data flow diagram 702A depicts an exemplary data flow for explicit rewriting of structured queries. The rewrite operation converts a received query (e.g., a raw SQL statement) provided (e.g., by a user) into, for example, an abstract syntax tree (AST), manipulates (rewrites) the AST, for example, by replacing attribute references, and finally transforms the AST representation back into a generated rewritten SQL statement. The rewritten statement is applied to a backing table as described herein. This approach provides a general solution, but requires the ability to rewrite the SQL dialect of the underlying system, for example, by transforming to / from an equivalent AST representation.

[0173] Now refer back to Figure 7B, data flow diagram 702A depicts an exemplary data flow for explicit rewriting of structured queries. The code provides automatic conversion of attribute references through rewrite rule definitions, such as through substitution triggers or updateable database views with extensive query rewrite capabilities. Changes are applied using raw SQL provided by the user and applied to versioned dataset views without parsing and explicit rewriting. Attribute mapping is done through underlying rewrite rules based on the rewrite rule definitions generated by the code. Such rewrite rule definitions are affected by changes to the backing table schema, particularly by DDL statements, but may also be affected by DML statements, such as column shadowing. The explicit rewrite implementation may require rewrite support from the underlying system and rewrite rule definitions from the dataset versioning extension code.

[0174] Now refer back to Figure 1 At 108, an updated versioned dataset is created in the backing dataset by applying at least one structured query to the specific version. The specific version can be selected from a plurality of versions of the backing dataset, for example, by a user. Alternatively, the specific version is automatically defined, for example, as a default value that can be adjusted. The specific version can be the last version of the dataset in the chronology of the dependent dataset and / or an intermediate dataset from which a new dependency branch is created. Applying the structured query to the specific version may create a new version of the specific version that is in addition to the existing specific version.

[0175] The term parent version is sometimes used interchangeably with the term specific version to indicate that a subsequent version (sometimes referred to herein as a new version) is created from a parent version by applying a structured query.

[0176] At the data level, a new lookaside table can be created as a copy of the parent dataset. Operations can then be applied directly to the new lookaside table. Both versions of the data are stored redundantly, and no auxiliary information is required within the two resulting lookaside tables. Alternatively, the new version can be integrated into its parent lookaside table.

[0177] At 110, the updated versioned dataset is stored in a backing dataset that stores multiple versions of the versioned dataset. The architecture of the backing dataset provides storage efficiency, for example, by using a single table and storing incremental changes to the original and / or previous versions rather than by storing each version separately.

[0178] The terms lookaside dataset and lookaside table are interchangeable, where the lookaside table can be implemented using other datasets that are not necessarily limited to tables.

[0179] Newly created datasets, i.e., updated versioned datasets, are copied to a lookaside table. The lookaside table can be newly created and / or can be a previously existing lookaside table. The lookaside table (e.g., column types) can be derived from the definition of the SQL query. Initially, the lookaside table does not necessarily store auxiliary information. Optional operations are applied directly to the lookaside table. The lookaside table may not be accessible to ordinary users.

[0180] The new version can be stored in an updated versioned dataset as a delta of a particular version, i.e., new and / or changed data is stored to reduce redundancy in the stored data, e.g., instead of storing a complete new version, there is a redundant copy of the data of the particular version for the new version.

[0181] Data, including previously stored versions of a particular version, remains unchanged and / or unaffected by the application of the structured query.

[0182] Optionally, the backing dataset is implemented as a table (eg, in an RDBMS), optionally as a single table, comprising rows and columns.

[0183] The original version of the versioned dataset is stored in its entirety. Subsequent versions of the versioned dataset are stored as incremental operations on the previous version by storing incremental rows and / or columns and by storing indications of the operations on the previous version in metadata of the versioned dataset catalog associated with the subsequent version, as described herein.

[0184] Optionally, a backing data set (e.g., a single table) that includes default values ​​indicating deleted data is compressed. The default values ​​indicate a mapping to the values ​​stored in the previous version. Default values ​​(e.g., null values) can be compressed efficiently, reducing storage requirements compared to storing redundant values.

[0185] At 112, a new fallback dataset is optionally created. The new fallback dataset includes a new dataset version created by applying one or more structured queries to a particular dataset. The new dataset version is used as the first dataset in the new fallback dataset. Additional versions are created based on the new dataset in the new fallback table.

[0186] Nesting of previous versions may result in decreased scan performance and / or increased cost of extracting separate versions of data from the existing backing dataset. New dataset versions are stored in new backing datasets to improve access and / or storage performance.

[0187] Alternatively, a decision to create a new dataset can be made using, for example, the following exemplary process. Configuration parameters indicate a tradeoff between storage cost and access cost for one or more versions of an updated versioned dataset. A cost model defines an overall cost for the new dataset based on the configuration parameters. A new fallback dataset is created for the one or more versions where the configuration parameters indicate an overall lower cost within the cost model.

[0188] The decision to start a new lookaside table is sometimes referred to as "materialization." The decision to "materialize" a specific version to which a structured query has been applied can be made manually and explicitly by the user, optionally individually for each dataset version, optionally at revision dataset creation time, for example, by setting configuration parameters for each versioned dataset. Alternatively, the decision to materialize can be automated, for example, by establishing a user-set configuration policy that defines the tradeoff between dataset version storage cost and dataset version access cost, as deep nesting of versioned datasets within a shared lookaside table can degrade scan performance due to the high cost of extracting data from the shared lookaside table for individual versioned datasets. Automation can use a cost model that evaluates the storage requirements versus access costs for a set of dataset versions and selects a separate storage model for each member of the set of dataset versions in one or more lookaside tables. Finally, code can transform between the storage representations of any dataset version at any time (automatically or manually triggered) to adjust the global storage and access cost characteristics of the versioned dataset repository as needed.

[0189] At 114, the versioned dataset catalog is updated. The versioned dataset can store metadata records, such as a record for each version of the dataset. The versioned dataset catalog includes a chronological history of structured queries applied and the lineage of each version of the backing dataset. This metadata can be stored as a structured query script, which can be used as documentation to define the provenance of the versioned dataset for collaborating users.

[0190] The versioned dataset catalog is updated to include at least one structured query associated with the updated versioned dataset and a lineage indicating that the updated versioned dataset includes a successor to the particular version.

[0191] An example representation of the lineage of corresponding versions stored in a version-controlled dataset directory shows that the dependencies between versions of the version-controlled dataset include: the immediate superior of the corresponding version, one superior among multiple superiors of the previous version, a root superior indicating the first version with no previous version, and the immediate superiors of two or more versions that are merged.

[0192] New metadata records can be created, and / or existing metadata records associated with existing versioned datasets can be updated. Optionally, the metadata record includes a given name for the new versioned dataset. The record can include the original SQL query defining the dataset and / or optional data manipulation statements, i.e., a chronologically ordered list of applied SQL statements. This SQL script can serve as documentation to define the source of the versioned dataset for collaborating users, and / or can later be used to define a list of operations for creating a "newer" version of the original versioned dataset based on the current state of the live data (e.g., the refresh operation described herein). The metadata can include the status of the versioned dataset (e.g., published / prepared / archived), optional user-defined textual annotations, and / or can include dependencies between multiple versioned datasets (e.g., direct ancestors, root ancestors, empty / not defined for the "first" version of a dataset) and / or relationships to backing tables. The creating user can be recorded as the owner of the dataset version. The selection / projection / mapping information of subsequent view definitions (if any) can be redundantly stored as metadata. Note that storing this information directly in the view definition may be sufficient.

[0193] A user-provided list of original operation statements can be stored in the metadata record. Overridden operations become obsolete after the operation. New datasets are marked as successors of the dataset version that serves as its parent. By following the chain of versioned dataset ancestors within the metadata, a complete list of defined operations (a chronological list of the initial query and all subsequent operations) is always available for any existing dataset version. Metadata provides complete documentation of the definition of each versioned dataset. Optional user-provided textual annotations documenting revisions can be stored in the metadata.

[0194] At 116, the dependency code is updated. For each version stored in the backing dataset, the dependency code indicates which data (e.g., rows and / or columns) of the updated versioned dataset matches which data (e.g., rows and / or columns) of the particular version, and which data (e.g., rows and / or columns) of the updated versioned dataset are deletions relative to the particular version.

[0195] A dependency encoding associated with the updated versioned dataset is updated to indicate which data of the updated versioned dataset matches the data of the specific version and which data of the updated versioned dataset is deleted relative to the data of the specific version.

[0196] If it doesn't already exist, a specific version of the affiliation code is generated. The affiliation code can be a separate data structure from the metadata and / or can be included in the metadata. After this first optional step, a new version of the affiliation code can be generated. Initially, the affiliation code is a copy of the affiliation information of the immediate superior.

[0197] In an exemplary implementation, the dependency encoding is implemented as a bitmap. In response to new rows and / or columns being added to an updated version-controlled dataset, the dependency encoding bitmap includes a Boolean value for each corresponding row, which is set to TRUE if and only if the corresponding row and / or column participates in the updated version-controlled dataset, and is set to FALSE otherwise. When the lookaside table is used to store only parent versions, a dependency bitmap for the parent dataset is generated so that all values ​​are set to TRUE. A bitmap is an efficient data structure (e.g., in terms of storage requirements and / or processing requirements) for indicating which rows and / or columns belong to which version.

[0198] Other implementations of dependency encoding are possible, such as encoding each version of the dependency in each row of array elements.

[0199] At 118, orphaned data can be deleted from the backing dataset. Orphaned data (e.g., rows and / or columns) in the backing dataset that is not associated with any version is identified. Orphaned data (e.g., rows and / or columns) are automatically deleted from the backing dataset. Deleting orphaned rows and / or columns improves storage efficiency by removing data that is not used by any version of the versioned dataset.

[0200] At 120 , a database view of the updated versioned dataset is generated by selecting data (eg, rows) of the backing dataset corresponding to the updated versioned dataset according to the dependency encoding and projecting other data (eg, columns).

[0201] A database view stores a mapping that indicates how to compute data (e.g., columns and / or rows) of an updated versioned dataset based on one or more previous versions of the data (e.g., columns and / or rows). A view of an updated versioned dataset defines instructions for applying computations to data (e.g., columns and / or rows) of one or more previous versions according to the mapping. Exemplary computations include rename, negate, type conversion, append, concatenate, any unary and n-ary mappings available in an RDBMS and user-defined mappings, and / or join.

[0202] Mappings can be used in the form of column and / or row calculations, for example, to avoid column name conflicts in backing datasets that may store data from multiple dataset versions that may have overlapping and conflicting column names. In some implementations, an internal canonical naming scheme for backing dataset columns is used (e.g., f0, f1, ...). For example, a mapping to the original user-provided column names, as defined by the initial structured query, can be provided within the view definition.

[0203] Computations can be defined by the host RDBMS, thereby enabling compatibility with and / or allowing the use of existing RDBMSs.

[0204] New database views may be created, and / or existing views may be updated. Views may be defined as named entities managed by the RDBMS and / or defined using a textual representation in the form of an SQL query to expose a versioned dataset created within the RDBMS using a given unique name. The view definition selects rows and columns from the corresponding backing table based on the associated dependency encoding. Optionally, mappings may be used in the form of column renaming to avoid column name conflicts in backing tables that may store data for multiple dataset versions that may have overlapping and conflicting user-defined column names. An exemplary implementation uses an internal canonical naming scheme for the backing table columns (e.g., f0, f1, ...) and provides a mapping within the view definition to the original user-provided column names, as defined by the initial SQL query.

[0205] When using a dedicated backing table (i.e., storing only new versions), the view is a simple view with no projections or selections, but may include mappings for column renaming. When using a common backing table (i.e., storing multiple versions), the view defines a choice of dependency encoding for the new versioned dataset, e.g., restricting data access to (e.g., only) rows in the backing table that participate in the current dataset version. Corresponding projections can be defined to restrict data access to (e.g., only) columns that participate in the current dataset version. Finally, mappings (e.g., renaming, negation, type conversion, addition, concatenation, join, etc.) can be defined for those columns of the updated versioned dataset that are computed from one or more columns of the backing table, e.g., using any unary and n-ary mappings available in the host RDBMS (including explicit user-defined mappings). When dependency encodings of parent snapshots are added during revision of the updated versioned dataset, views of the parent version (i.e., a specific version) can also be updated to use the choice of dependency encoding and projection vector.

[0206] Views can provide controlled read access to backing tables through permissions defined and enforced within the RDBMS. Optionally, views may initially be accessible only to the owner of the updated versioned dataset, but can be granted to other users through the host RDBMS's permissions system. Updated versioned datasets can be created in the "Prepared" state, emphasizing that the updated versioned dataset is currently in progress. An alternative implementation may create the updated versioned dataset in the "Published" state. Other users with the corresponding permissions can browse the metadata and discover the versioned dataset, along with full documentation of its provenance, and access the data with authorization from the individual owner.

[0207] At 122, one or more features described with reference to 104-122 may be repeated, for example, to process other structured queries by the same user and / or different users. Other users (e.g., with corresponding access rights) may browse the shared metadata and / or discover the versioned dataset and the complete documentation of its source and / or access the data if authorized (e.g., by the owner of the corresponding dataset version).

[0208] Now refer back to Figure 3 , Figure 3 Some examples of structured queries represented by features. Figure 4 Describe intuitively Figure 3 For each feature, an overview is provided first, followed by additional details. Figure 4 As shown in 400 , the solid circles represent data sets created by applying a structured query (eg, SQL) to an existing data set.

[0209] At 302, a refresh operation is performed, which Figure 4 402. A refresh operation can be used later to create a "newer" version of the original (or other lineage) version-controlled dataset based on the current state of the live data (or other updated data).

[0210] It should be noted that the operations described herein are exemplary and do not necessarily encompass the full set of possible operations. For example, the modify operation may be optional in some implementations and may be omitted without compromising the general applicability of the present invention. Furthermore, the names of the various operations (e.g., create, revise, etc.) and / or specific states (e.g., published, prepared, archived) provided herein are exemplary and not necessarily limiting.

[0211] The following is an exemplary implementation of a refresh operation: re-execute a historical structured query stored in a version-controlled dataset catalog that has been applied in chronological order to create an updated version of a particular version of a plurality of versions of the version-controlled dataset based on current real-time data. The updated version is stored in the version-controlled dataset. The version-controlled dataset catalog is updated with the lineage and an indication of the re-execution of the updated version.

[0212] Note that refreshing the history of a versioned dataset, for example if the schema of the live data has changed, may result in conflicts that may need to be resolved. The metadata of the merged dataset reflects the list of resolved conflict-free operations.

[0213] At 304, an insert operation is performed to insert the new data into a specific version. Storage efficiency is provided by storing incremental new data for the new version, rather than by storing the entire new version. The old version is not affected and is maintained using metadata.

[0214] An insert operation can be used to insert new rows and / or columns into a specific version. The backing dataset is updated to include the new rows and / or columns. The dependency encoding is updated to indicate that the new rows and / or columns are associated with the updated versioned dataset. The new rows and / or columns are not associated with other versions. A view of the updated versioned dataset, created by selecting the rows of the backing dataset corresponding to the updated versioned dataset and projecting the columns, includes the new rows and / or columns.

[0215] At 306, an operation is performed to delete data (e.g., rows and / or columns) from a particular version. The dependency encoding is updated to indicate that the rows and / or columns of the particular version are excluded from the updated versioned dataset, wherein the rows and / or columns are not deleted in the backing dataset, and the dependency encoding is updated to indicate that the deleted rows and / or deleted columns are not associated with the updated versioned dataset, and the deleted rows and / or deleted columns remain associated with other versions.

[0216] When accessing a previous version of a dataset, the actual data for deleted rows and / or columns remains in the backing table. Views of the updated versioned dataset created by selecting the rows and / or projected columns of the backing dataset that corresponds to the updated versioned dataset do not include the deleted rows and / or columns.

[0217] At 308, an operation is performed to change the data of a particular version (e.g., the values ​​in rows and / or columns). Data not affected by the change is retained in a backing table and used when accessing previous versions of the dataset. The data affected by the changed value (e.g., rows and / or columns) is copied. The backing dataset is updated to include the copy. The value on the copy has changed. The dependency encoding is updated to indicate that the copy with the changed value is associated with the updated versioned dataset. The copy with the changed value is not associated with other versions. The unchanged value in the copy associated with the updated versioned dataset is stored as a default value that indicates a mapping to the corresponding data stored in the particular version. A view of the updated versioned dataset is presented by replacing the default value with the corresponding data according to the mapping. The default value can be compressed efficiently so that only one copy of the actual value is stored instead of multiple copies.

[0218] An exemplary row update operation includes a combination of INSERT and DELETE. All rows in the parent version affected by the update (i.e., the original rows) are copied (e.g., INSERT) to the current version (i.e., the copied rows). The original rows are deleted (e.g., DELETE) from the current version by updating the dependency code. The update is then applied to the copied rows in the current version. The dependency code helps ensure correct access to the corresponding rows without requiring updates to the parent dataset or the view definition of the current dataset.

[0219] An exemplary update operation for a column includes: creating a copy of the parent version column (i.e., the original column) affected by the update operation, rewriting the update operation so that the copy is updated instead of the original column, and updating the dataset view projection of the current version to access the copied column instead of the original column.

[0220] An exemplary general dependency encoding process for update operations includes encoding the update operation using encodings of existing and new columns with functional mappings in the dataset view definition. In the example (column shadowing), columns (target columns) are created using the column definitions of the columns affected by the parent version update (original columns) and populated with default values ​​(e.g., NULL for undefined). Update operations are redirected to populate the target columns with the updated values ​​instead of overwriting the values ​​in the original columns. The dataset view projection of the current version is updated to access the first target column, and if the value of that target column is the default value (e.g., NULL for undefined), the original column is accessed (e.g., COALESCE(target, original)).

[0221] At 310, an operation of amending (also referred to herein as amending) and / or revising (also referred to herein as revising) one of the versions stored in the backing dataset is performed. The modification of one of the versions is visually depicted at 404. An existing dataset can be modified rather than creating a new version. In contrast, to help understand the difference between amending and revising, in Figure 4 A revision operation is visually depicted in 406 of the present invention, which is performed on a specific version by applying a structured query to the specific version to create a new version (e.g., as shown in reference Figure 1 Said, wherein reference Figure 1 The features described may be generally described for revision operations).

[0222] An exemplary modification operation includes: selecting one of the versions stored in a backing dataset; verifying that the selected version has no dependent versions, for example, verifying that the selected version is a terminal node in a version dependency graph; applying at least one structured query to the selected version; updating metadata of a versioned dataset catalog associated with the selected version to further include at least one structured query applied to an existing chronologically applied historical structured query; and updating a view of the selected version based on the applied at least one structured query.

[0223] Additional details are now provided regarding exemplary modification and / or revision operations. The dataset versioning extension code (e.g., API) allows a user to modify an existing "prepared" versioned dataset, which is also owned by that user. Modifications cannot be made to datasets that already have dependent versions. This means that only the dataset version at the tip of any branch in the version graph can be modified. Internal datasets can be revised. Datasets in the "published" state cannot be modified, as these datasets are intended to be "shared" with other users. Similarly, "archived" datasets cannot be modified.

[0224] Technically, the modify operation is very similar to the revision operation, but instead of creating a new version, it updates and extends an existing version. The user provides a unique name for all dataset versions that do not currently have existing dependencies, along with a list of data manipulation statements. Optionally, the user provides a text comment describing the rationale / purpose of the modified dataset version. Any qualified dataset version can be modified repeatedly. Other users can create dependent dataset versions (revisions) of a given dataset version at any time, thereby prohibiting modification of that specific dataset version. In this case, the user can create a revision as a new branch instead of a modification.

[0225] At the data level, the code continues to apply data operations, producing exactly the same results as if the modified operations had been appended to the data operations list of the original revision step.

[0226] At the metadata level, the original operation statements provided by the user are appended to the chronologically ordered list of operation statements in the existing metadata recorded in the modified dataset version. An optional user-provided text comment documenting the modification is also recorded in the metadata. If necessary, the view definition is updated; for example, projections will reflect added or deleted columns, or mappings will define column renames and access to computed data. The dataset version's name, view names, and all established access permissions remain unchanged. The modified dataset version remains in its original state, as prepared.

[0227] At 312, an operation of sampling and / or partitioning at least a subset of a particular version is performed. For example, the sampled dataset can be used to create a different dataset for training and / or validating an ML model.

[0228] The sampling and / or partitioning operations may be performed by creating an updated versioned dataset comprising the at least one subset. A versioned dataset catalog of the updated versioned dataset includes a lineage indicating that the at least one subset of the updated versioned dataset is derived from a particular version.

[0229] Additional details of an exemplary sampling operation are now provided. The dataset versioning extension code (e.g., API) allows a user to sample any read-accessible dataset version (i.e., a revisable dataset version) to create one or more subsets and persist these subsets as individually named dataset versions, each with the original dataset version as its common ancestor. The user provides a unique name for the parent dataset version to sample and, for each dataset, provides sampling instructions to sample, for example, the following:

[0230] ● Size specifications, such as the percentage of the sample relative to the parent dataset size (i.e., a number in the interval [0.0...1.0]), or the absolute number of rows in the sample (i.e., a number in the interval [1...number of parent rows]).

[0231] ● Name specification, such as the full name or a suffix to be appended to the parent dataset name to generate a unique name for each sampled dataset

[0232] ● Sampling specifications: e.g., non-deterministic, deterministic, stratified, etc.

[0233] Text annotations to record sampling revisions

[0234] At the data level, sampling can be implemented by copying a subset of rows (selected according to the user's sampling specifications) to a new backing table. Alternatively, a dataset sample can be efficiently stored in a common backing table along with its parent. For example, this can be achieved by copying the parent dataset's affiliation information and then removing the participation of each row from the copy according to the user's sampling specifications. Both of these exemplary approaches create a physical persistent representation of the dataset sample. As another alternative, the dataset sample can be created as a purely logical entity that is calculated directly from the parent dataset, such as by selection, without the need for modification / representation at the data level.

[0235] At the metadata level, each dataset sampling represents a new, separate dataset version, with its own metadata record. A new dataset is marked as a successor to the dataset version that serves as its parent. Sampling instructions are recorded similarly to defining SQL queries and data operations, so following the chain of predecessors within the metadata always provides a complete list of defined operations for any existing dataset version (a chronologically ordered list of the initial query, all subsequent operations, and sampling instructions). The metadata provides complete documentation of the definition of each versioned dataset. Optional user-provided text comments documenting revisions are also recorded in the metadata.

[0236] A specialized view definition is created based on the method selected at the data level, acting as a view that extracts data from the corresponding backing table. As an alternative to physically sampling a persistent dataset, a view definition can also retrieve data directly from the parent dataset, so that the user's sampling instructions are reflected in the view definition, defining the appropriate row selection from the parent dataset. Projection and mapping definitions are typically inherited from the parent dataset. The logical method of sampling is expressed at the metadata and view levels, but not at the data level.

[0237] Creates a dataset sample in the "Prepared" state. Alternative implementations can inherit the original state of their parent, either Prepared or Published.

[0238] At 314, a random permutation operation is performed on the specific version of the data, i.e., an operation that changes the order of the data. For example, random permutation of a dataset can be used to create different datasets for training and / or validating an ML model.

[0239] The shuffle operation may be performed by creating an updated versioned dataset including shuffled data. The shuffled data is implemented as at least one of a mapping of data from a specific version to a new order in the updated versioned dataset and a copy of the data from the specific version to the new order in the updated versioned dataset. A versioned dataset directory for the updated versioned dataset includes instructions indicating that the updated versioned dataset with the shuffled data is a lineage derived from the specific version and instructing to shuffle the data from the specific version to create the updated versioned dataset.

[0240] Additional details of an exemplary shuffle operation are now provided. The dataset versioning extension code (e.g., API) allows a user to shuffle any read-accessible dataset version to create one or more shuffled permutations and persist these shuffled permutations as individually named dataset versions, each with the original dataset version as its common ancestor. The user may provide a unique name for the parent dataset version to shuffle, and for each dataset shuffle instruction, provide a shuffle instruction that shuffles, for example, the following:

[0241] ●Name specification, such as the full name or a suffix to be appended to the parent dataset name, used to generate a unique name for each randomized dataset.

[0242] ● Random permutation specifications, such as random non-determinism, random determinism, lexicographic order of (possibly mapped) column attributes, etc.

[0243] ●Record text annotations for random order revisions.

[0244] At the data level, the potential for representing random permutations is limited because typical RDBMSs cannot enforce or guarantee deterministic row read order for relations (datasets), a consequence of the underlying relational algebra that typically operates on unordered sets. Therefore, creating replicas in separate lookaside tables can only be used to create a physical representation of random, non-deterministic random permutations. General-purpose deterministic random permutations typically use shared lookaside tables. By reusing parent dependency information (if present, eliminating the need for replicas in any case), it is possible to create a random permutation of versioned datasets based on the parent. If the user specifies the random permutation order as a lexicographic ordering of mapped column attributes, the desired random permutation order can be established through explicit row reordering (sorting) according to these instructions (to be defined at the view level). Alternatively, the lookaside table can be extended with an auxiliary sequence column (as part of the dependency and order encoding of the dataset version) that defines the sequential position of rows within the random permutation. Populating the auxiliary sequence column with values ​​allows the system to establish the desired row order by sorting on the auxiliary sequence column. The view definition then leverages the dependency and order encoding to provide a stable ordering through sorting.

[0245] At the metadata level, each dataset shuffle represents a new, separate dataset version, with its own metadata record. A new dataset is marked as a successor to the dataset version that serves as its parent. Shuffle instructions are recorded similarly to defining SQL queries and data operations, so following the chain of predecessor versions within the metadata always provides a complete list of defined operations (a chronological list of initial queries, subsequent operations, sampling, and shuffle instructions) for any existing dataset version. The metadata provides complete documentation of the definition of each versioned dataset. Optional user-provided text comments documenting revisions are also recorded in the metadata.

[0246] Successive revisions, modifications, and sampling of the randomized dataset preserve the established order, such that the most recent sequential instruction in a chronologically ordered list of operations applies to all dependent dataset versions. If a defined operation list contains multiple randomization instructions, the most recent randomization supersedes the previous ones. Alternative implementations may use a scheme to merge non-conflicting randomization instructions.

[0247] At the view level, dataset shuffles use the same selections (i.e., encoding dependencies with respect to the parent, if any), projections, and mappings as their parent dataset. Furthermore, shuffles introduce the concept of order into the view definition. Shuffle order can be defined lexicographically on any combination of (possibly mapped) user-defined columns, or on auxiliary columns (dedicated to versioned dataset shuffles) for sequence encoding. The order is typically enforced by a corresponding directive (ORDER BY) in the view definition, which establishes the desired order through sorting.

[0248] The view provides controlled read access to the backing table through permissions defined and enforced within the RDBMS. The initial permission settings for a dataset's random permutation are inherited from the parent dataset.

[0249] Dataset shuffles are created in the Prepared state. Alternative implementations can inherit the original state of their parent, either Prepared or Released.

[0250] At 316, a clear operation of the specific version is performed. Figure 3 This is shown visually in 408. The purge operation may attempt to free up occupied storage capacity of the purged dataset while keeping the integrity (eg, data and history) of possible dependent dataset versions intact and unaffected.

[0251] An exemplary purge operation may be performed by: deleting data from a backing dataset that is exclusive to a particular version and not present in any dependent datasets; deleting metadata from a versioned dataset catalog that is exclusive to a particular version and not present in any dependent datasets; and updating metadata of the versioned dataset catalog or at least one dependent dataset by changing a dependency from the purged version to an earlier version on which the purged version depends.

[0252] Additional details of an exemplary purge operation are now provided.

[0253] Dataset versioning extension code (e.g., API) allows users to purge existing dataset versions (published, prepared, or archived), which are also owned by the user. The purge operation removes the specific dataset version from the system, rendering it inaccessible. Furthermore, the purge operation attempts to free up the storage capacity occupied by the purged dataset while leaving the integrity (data and history) of any dependent dataset versions intact and unaffected.

[0254] At the data level, if there are no dependent dataset versions, all data associated with the purged dataset version can be deleted. The same applies if dependent datasets exist but are stored in separate lookaside tables. Otherwise, either (A) data dependencies can be resolved by recoding the dependency information in the common lookaside table, and then the data can be deleted, or (B) dependent datasets can be relocated to different lookaside tables.

[0255] (A) This can be achieved, for example, by deleting by dependency: all rows and columns belonging exclusively to the purged dataset are deleted from the common lookaside table. This automatically preserves all rows and columns marked as shared between dependent dataset versions in the dependency encoding of the shared lookaside table. Dependency information that only serves the purged dataset may also be deleted or re-encoded (depending on the implementation). Thus, the purged dataset is integrated into all its successor datasets.

[0256] (B) can be achieved, for example, by relocation: the dataset versioning extension moves all dependent datasets (possibly recursively) to a separate new lookaside table, or dependent datasets are pushed down to a separate existing lookaside table. As a result of this reorganization, the dataset to be purged has no dependent datasets within the same lookaside table and can be safely deleted. Once the data integrity of the dependent datasets has been established, all rows and columns that are referenced only by the purged dataset can be deleted from their lookaside tables. Dependency information that only serves the purged dataset can also be deleted or recoded (depending on the implementation). The purged dataset is thus integrated into all its successor datasets.

[0257] When reorganizing storage during a purge operation, the code is free to choose between any combination of recoding and relocation of each dependent dataset. It is also possible to reverse-relocate dependent datasets from separate lookaside tables to shared lookaside tables, and recode dependencies accordingly. The code may aim to meet configuration goals, such as through user-set configuration parameters, or by applying the policy-driven automatic storage model selection introduced previously, to satisfy a given trade-off between storage efficiency and data retrieval performance for all individual dataset versions affected by the purge operation, not just their immediate descendants.

[0258] At the metadata level, each current ancestor of the purged dataset (which may not exist after the merge operation and may be one or more) becomes a direct ancestor of all dataset versions that directly depend on the purged dataset. To maintain the historical integrity of operations that define dependent dataset versions, the list of operations originally associated with the purged dataset is placed before the list of operations of all direct successors, so that following the chain of ancestor versions in the metadata can always provide a complete list of defined operations for any existing dataset version (a chronologically ordered list of the initial query and all subsequent operations). Similarly, the descriptive user annotations originally associated with the purged dataset are placed before the user annotations of all direct successors. Finally, the metadata record corresponding to the purged dataset is deleted.

[0259] The views corresponding to the cleared datasets are deleted, and therefore all existing access permissions are revoked.

[0260] Purged datasets are deleted without leaving traces at the data, metadata, and view levels. Accordingly, these datasets have no state.

[0261] Other users with the appropriate permissions might browse the metadata and discover that the purged dataset no longer exists, making it inaccessible. The unique name of the purged dataset can be reused to name new datasets for any user.

[0262] At 318, an operation is performed to merge the first version with the second version. The first version and the second version are derived from a common superior version. The merge operation is performed at Figure 4 The 410 is intuitively depicted.

[0263] A merge operation can be performed by integrating operational operations applied to the first version with operational operations applied to the second version to create a merged set of operational operations. An exemplary merge operation includes applying the merged set of operational operations to a common superior version. An indication of lineage derived from the first version and the second version and the merged set of operational operations is stored in metadata of the versioned dataset catalog associated with the updated versioned dataset.

[0264] Additional details of an exemplary merge operation are now provided. The dataset versioning extension code (e.g., an API) allows a user to merge any two read-accessible dataset versions (possibly created by different users) with separate histories into a new version. The user provides unique names for the two dataset versions, such as LEFT and RIGHT. Optionally, the user provides a text comment describing the rationale / purpose of the merged datasets. In essence, the operation extracts a chronologically ordered list of defined operations, starting with the most recent common ancestor of LEFT and RIGHT. If no common ancestor exists, the merge operation fails. The effect of a successful merge operation is equivalent to a revise operation, using the operation list of operations generated by integrating the two previously extracted operation lists into a common linear history, and creating a new revised dataset version by applying the integrated operation list to the common ancestor. The merged dataset version becomes the successor of LEFT and RIGHT in the dataset version graph.

[0265] At the data level, the process is comparable to the general revision process, e.g., you choose a storage model where new revisions are stored in a different or shared backing table.

[0266] At the metadata level, the difference from revisions is that the resulting dataset version is marked as having two ancestors (LEFT and RIGHT) in its metadata. The list of defined operations for the merged dataset (a chronologically ordered list of initial queries, subsequent operations, sampling, and shuffle instructions) is defined as a properly serialized list of linear operations resulting from integrating two parallel branches leading from the most recent common ancestor to LEFT and RIGHT. An optional user-provided textual annotation documenting the revision is also recorded in the metadata.

[0267] Merging versioned dataset histories, for example where the schemas of LEFT and RIGHT have diverged, may result in conflicts that require conflict resolution (discussed later). The metadata of the merged dataset reflects the list of resolved conflict-free operations, which is used as the new history of the merged dataset, replacing the divergent histories of the merged dataset's two ancestors, LEFT and RIGHT.

[0268] At the view level, the process is comparable to the usual revision process, where different views are created depending on the selected storage model.

[0269] The view provides controlled read access to the backing table through permissions defined and enforced within the RDBMS. The view is initially accessible only to the owner of the merged dataset, but can be granted to other users through the host RDBMS's permissions system.

[0270] Merged datasets are typically created in the "Prepared" state, emphasizing that they are currently a work in progress. Alternative implementations can create these datasets in the "Published" state.

[0271] At 320, an operation is performed to change the state of the updated dataset version. Dataset versions can be assigned states. In an exemplary implementation, a dataset version can be created in a "Released" state, emphasizing the importance of the initial dataset version as the starting point of the version graph. Alternative implementations include a "Prepared" state and other states described herein.

[0272] An example publish operation will now be described. Dataset versioning extension code (e.g., an API) allows a user to publish an existing dataset version (currently in the "archived" or "prepared" state), which is also owned by that user. The user provides a unique name for the dataset version to be published. The publish operation is a purely metadata operation that changes the state of a given dataset version. The "published" state highlights datasets that are ready to be shared with other users for collaboration (e.g., datasets for which data governance processes have been completed). Data, views, and permissions are unaffected by the publish operation.

[0273] An example archive operation is now described. Dataset versioning extension code (e.g., an API) allows a user to archive an existing dataset version (currently in the "Published" or "Prepared" state) that is also owned by this user. The user provides a unique name for the dataset version to be archived. The archive operation is a purely metadata operation that changes the state of a given dataset version. The "Archive" state highlights datasets that are outdated (e.g., should not be used by collaborating users). Data, views, and permissions are unaffected by this operation.

[0274] At 322, an operation is performed to extract a patch from a chronologically ordered list of applied operations. Figure 4 416 . The user can provide unique names for two dataset versions, such as START and END, where START is an ancestor of END. The result will expose a textual list of sorted operations to the user, for example, for inspection and / or editing. Patches can be directly applied to revise or modify an existing revisable dataset version by passing the patch as an input parameter to the revise or modify function, respectively. Applying a patch may result in conflicts, which may need to be resolved. Patching is a purely read operation on metadata and does not affect the data, metadata, views, permissions, or the state of the dataset versions involved.

[0275] At 324, a rebase operation is performed. The patch operation is Figure 4418 . The dataset versioning extension code (e.g., an API) allows a user to rebase the history of any read-accessible dataset version. The user may provide three unique names for the dataset versions, such as OLD, NEW, and HEAD, where OLD is an ancestor of HEAD. The rebase operation extracts a chronologically ordered list of structured query operations from OLD to HEAD and applies those lists to NEW. The effect of a successful rebase operation is equivalent to a revision operation, creating a new revised versioned dataset using the generated list of operations, and then purging all versioned datasets from OLD (exclusive) to the original HEAD (inclusive). Rebasing the dataset history may result in conflicts, for example, where the schemas of OLD and NEW are different, and conflicts may need to be resolved. The metadata of the rebased dataset reflects the resolved list of conflict-free operations.

[0276] At 326, conflict resolution operations can be performed, for example, to resolve conflicts that may arise from implementing some of the operations described herein, such as resolving incompatible schemas when merging two dataset histories. Conflict resolution operations can include automatic conflict detection (e.g., type, arity, column names, etc.) and / or allow manual resolution by editing patches. Semi-automatic resolution strategies can resolve conflicts when merging histories, such as preserving existing values ​​or overwriting values.

[0277] Now refer back to Figure 8 According to some embodiments, data flow 802 depicts a merge operation on a text file in CSV format that results in conflicts, while data flow 804 depicts a merge operation without conflicts using relational algebra.

[0278] Now refer back to Figure 6 , at 602, a user accesses a dataset versioning interface (API) 604. The user can call stored procedures that make up the API to support dataset versioning in an RDBMS. For example, as described herein, the dataset versioning interface (API) 604 provides functionality tailored for efficient storage, collaboration, and automation on management of versioned datasets. The dataset versioning extension exposes a set of executable functions for managing versioned datasets. The actual user interface can be customized to provide convenient support, such as for use cases (a) deep branching, (b) flat branching, (c) scientific mode, (d) data governance mode, or other similar use cases. The implementation of the dataset versioning extension described herein can be independent of the user interface. General dataset versioning can be integrated into an RDBMS as a common and efficient method for storage management of versioned datasets, rather than necessarily a specific API.

[0279] For example, as described herein, the dataset versioning extension (code) 606 provides general dataset versioning functionality using existing RDBMS infrastructure. The dataset versioning extension code 606 can be an extension of the host RDBMS that can be added to an existing production RDBMS at any time to equip the RDBMS with dataset versioning capabilities. The extension installs executable functionality, i.e., code, for managing versioned datasets and creates an empty repository to store versioned datasets. The dataset versioning repository 608 can define a namespace within the RDBMS for storing relational data representing datasets with versioning information. RDBMS users may not have direct access to data stored in the repository, and access may be restricted by the RDBMS's permission system. Since all data is stored in the RDBMS, the dataset versioning repository integrates the full recovery capabilities of the RDBMS, and the same permission system can be used for live data and versioned data.

[0280] For example, as described herein, the dataset version control repository 608 collects, maintains, and / or utilizes relevant data and / or metadata on the version-controlled dataset storage 610 and / or catalog 612 for dataset lineage and / or reuse in collaborative workflows, and to improve productivity through automation. For example, as described herein, the version-controlled dataset storage (data) 610 provides space-efficient storage management for the complex dependencies between the redundant parts of the dependent versions included in the version-controlled dataset. The version-controlled dataset storage can include one or more fallback tables, each of which optionally stores version-controlled relationship data in a traditional RDBMS, close to where the real-time dataset exists and evolves, for example, as a traditional relationship in the RDBMS system. The fallback table is a physical representation of the version-controlled dataset and can be managed by the dataset version control extension code. The fallback table is defined and managed by the dataset version control extension code. All read / write access rights of ordinary users may be revoked. Each fallback table can be used to store one or more versions of a single version-controlled dataset. Lookaside tables can be created using the full expressive power of SQL, based on user-provided dataset definitions in SQL (e.g., "CREATE TABLE AS SELECT <…>"), using any combination of live datasets and historical data from existing versioned datasets stored within or accessible to the system. The data types of the lookaside table columns are derived from the input data, and if applicable, other properties are inherited from the input data, such as check constraints (e.g., NOT NULL), unique constraints, collations, and similar column definition attributes. Lookaside tables that store more than one dataset version also utilize dependency encoding (e.g., dependency bitmaps) to store auxiliary attributes, allowing code to associate any data row within the lookaside table with one or more dataset versions that share the lookaside table. The minimum requirements for implementing relational lookaside tables in a given host RDBMS may be that the host system is capable of storing wide tables, dynamically adding columns to existing lookaside tables, and efficiently handling sparsity (null value regions) within the lookaside table. Therefore, many implementations of relational tables are suitable. Backing tables can leverage existing storage technologies for compact storage and scalable, efficient retrieval, such as compressed (optionally distributed) columnar storage, which also allows cheap addition and removal of columns from existing tables. Alternative implementations can use the cloud-native wide column storage of a given host system (HBase / BigTable / DynamoDB, Azure Tables, Hypertables, etc.), with similar compression and flexible column management capabilities, but backed by the scalable throughput and abundant capacity of cloud storage services.

[0281] For example, as described herein, the versioned dataset catalog (metadata) 612 provides read access for browsing available datasets in the repository, for collaboration and reuse of data and operations on versioned datasets.

[0282] Dependency codes may be stored in the versioned dataset catalog 612 and / or in a separate data structure. Dependency codes may define auxiliary columns stored in a backing table that includes more than one version of a particular versioned dataset. Dependency codes are not necessarily exposed to users in the versioned dataset view or elsewhere; they may be used for internal purposes. One exemplary implementation of dependency codes is a dependency bitmap, i.e., a Boolean attribute (NOT NULL) per dataset version stored in the backing table, where for each row that participates in the version corresponding to the bitmap, the corresponding Boolean value is set to TRUE. Otherwise, it is set to FALSE. Alternative implementations of dependency codes may use a single attribute or a combination of multiple attributes of different data types, such as INTEGER, BINARY, VARBINARY, VARCHAR, BOOL[], or INTEGER[], to encode dependencies for multiple versions. Versioned dataset views 614 use dependency codes to filter out rows that do not participate in a single version of the versioned dataset represented by the view.

[0283] In addition, the dependency encoding can also define a specific order for each dataset version of the version-controlled dataset. The order can be stored in the version-controlled dataset directory 612 and / or stored in a separate data structure, for example, in an auxiliary column in a lookaside table.

[0284] For example, as described herein, a versioned dataset view 614 provides controlled exposure of a versioned dataset in the namespace of a live database. Access permissions may apply. The RDBMS can (optionally continuously) update live datasets 616, such as ML training datasets, production data that serves stored business logic. Full SQL interaction with versioned data (read-only) and / or live data (read / write) can be supported. For example, as described herein, at 618, data from live datasets can be merged to create a versioned dataset. A versioned dataset view can expose a separate version of a versioned dataset as a relational view. These views can be defined and managed by dataset versioning extension code. For example, definitions include mappings, projections, selections, and sorting. Ordinary users may be granted read-only access to a view and may not necessarily be able to manipulate the view definition. Such a view allows controlled access to a corresponding backing table. The view provides a mapping of backing table columns to user columns. A mapping can refer to a simple mapping of column names, and / or to any functional combination (expressible in SQL) of one or more backing table attributes to computed columns (e.g., type conversion, arithmetic, joins, cascades, etc.). This view provides projection by hiding backing table columns that do not participate in the single version of the versioned dataset represented by the view. Projection can be strictly enforced by the projection list of an immutable view definition. This view provides selection by hiding backing table rows that do not participate in the single version of the versioned dataset represented by the view. Selection can be enforced using dependency encoding predicates in the immutable view definition. This view provides order by enforcing a specific per-row read order, which can be defined for each version of the versioned dataset within the associated dependency encoding. Such a view can provide read-only access to the immutable backing table to the owner of the versioned dataset (e.g., the ordinary user who created the particular versioned dataset) for controlled access. The owner's read-only permission may be grantable, allowing the owner to share the versioned dataset with other RDBMS users. These views can allow arbitrary, seamless querying and data migration between a live dataset and multiple historical, versioned datasets, while providing full compatibility with the type system.

[0285] Now refer back to Figure 10 , the data version control repository 1002 can be defined and / or managed by code, which is optionally an add-on to an existing RDBMS, sometimes referred to herein as a versioned dataset extension. Lookaside tables T1 1004 and T2 1006 can store restrictions and / or internal names 1008 that are denied access to ordinary users. Public names 1010 can provide read-only access to ordinary users, which can be granted by the dataset version owner.

[0286] Lookaside table T1 1004 stores a single dataset version V1, e.g., the initial version of the dataset. Because there is only one version, dependency encoding is not necessary. Users can be granted access to a versioned dataset view 1012 of lookaside table T1 1004, generated by mapping the columns of lookaside table T1 without dependency encoding and / or without changing the order.

[0287] Backing table T2 1006 stores three versions of the dataset, denoted as V2, V3, and V4. V2 is given a dependency code 1014, V3 is given a dependency code 1016, and V4 is given a dependency code 1018. A user is granted access to dataset view V2 1020, which is created by mapping and projecting T2 columns using selects according to and / or the order defined by dependency code 1014. Dataset view V3 1022 is created by mapping and projecting T2 columns using selects according to and / or the order defined by dependency code 1016. Dataset view V3 1024 is created by mapping and projecting T2 columns using selects according to and / or the order defined by dependency code 1018.

[0288] Now refer back to Figure 11 , data flow 1102 depicts an exemplary ML data flow for preparing and / or managing multiple versions of a version-controlled dataset for reproducible training and testing of ML models. For example, user 1 performs data cleansing, user 2 performs snapshot specialization, and user 3 performs data interpolation by corresponding structured queries on different versions of the version-controlled dataset, as described herein.

[0289] For example, in an exemplary ML process, as depicted in data flow 1102, several versions of a dataset are created from raw data through an elaborate and often repeated dataset preparation process (e.g., data curation / cleaning / imputation), and then prepared for training a specific model (e.g., feature extraction / dataset specialization). Typically, multiple data scientists participate in these processes, and at least some embodiments described herein support this collaboration by maintaining a centralized repository of individual contributions to encourage sharing and reuse of existing work units. ML datasets (e.g., the exemplary implementation of a version-controlled dataset described herein) can be annotated with a unique name (version label), a status (e.g., prepared, published, archived, purged), and documentation as textual annotations.

[0290] In at least some embodiments, a detailed record of the complete lineage of each versioned dataset is maintained, from the original data through all intermediate versions, including the defined transformations of each version relative to its immediate predecessor, up to its pre-determined state. This complete lineage provides reproducible model training and / or provides data provenance for trained ML models by preserving the training dataset as effectively unchanged.

[0291] Other systems, methods, features and advantages of the present invention will be or become apparent to one skilled in the art upon examination of the following figures and detailed description. It is intended that all such other systems, methods, features and advantages be included within this description, be within the scope of the present invention, and be protected by the following claims.

[0292] The descriptions of various embodiments of the present invention are for illustrative purposes only and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles, practical applications, or technological advancements of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein, as compared to existing technologies on the market.

[0293] It is expected that many related RDBMSs will be developed during the life of the patent expiring on this application, and the scope of the term RDBMS is intended a priori to include all such new technologies.

[0294] As used herein, the term "about" refers to ±10%.

[0295] The terms "comprises," "comprising," "includes," "including," "having," and variations thereof mean "including but not limited to." This term encompasses the terms "consisting of" and "consisting essentially of.

[0296] The phrase "consisting essentially of" means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0297] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0298] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0299] As used herein, the word “optionally” means “provided in some embodiments and not provided in other embodiments.” Any particular embodiment of the present invention may include a number of “optional” features, unless such features are in conflict.

[0300] In this application, various embodiments of the present invention may be presented in range format. It should be understood that the description in range format is for convenience and brevity only and should not be construed as a fixed limitation on the scope of the invention. Therefore, the description of a range should be considered to have specifically disclosed all possible subranges and individual numerical values ​​within the range. For example, a description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within the range such as 1, 2, 3, 4, 5, and 6. This applies regardless of how broad the range is.

[0301] When a numerical range is indicated herein, any recited number (fractional or integer) within the indicated range is included. The phrases "a range between a first indicated number and a second indicated number" and "a range from a first indicated number to a second indicated number" are used interchangeably herein to include the first indicated number and the second indicated number and all fractions and integers therebetween.

[0302] It will be appreciated that, for the sake of brevity of description, certain features of the present invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity of description, various features of the present invention described in the context of a single embodiment may also be provided individually or in any suitable subcombination or as any suitable other embodiment of the present invention. Certain features described in the context of individual embodiments are not considered essential features of those embodiments, unless the embodiment would be inoperable without those elements.

[0303] It is the applicant's purpose that all publications, patents and patent applications mentioned in this specification are incorporated herein by reference in their entirety, as if each individual publication, patent or patent application were specifically and individually indicated when mentioned to be incorporated herein by reference. In addition, citation or identification of any reference in this application should not be construed as an admission that the reference is available as prior art for the present invention. With respect to the use of section headings, section headings should not be construed as necessary limitations. In addition, the entire contents of any one or more priority documents of this application are incorporated herein by reference.

Claims

1. An apparatus (204) for managing a version-controlled dataset (216B), characterized in that include: A processor (202) and a non-transitory storage medium (206) carrying information (206A) that, when executed by the processor, causes the processor to: Receiving at least one structured query (212A), the at least one structured query indicating an instruction to operate on a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backing dataset (216A); creating an updated version-controlled dataset in the backing dataset by applying the at least one structured query to the specific version; storing the updated version-controlled dataset in the backing dataset, wherein the updated version-controlled dataset stores incremental changes to a particular version without generating redundant copies of the data for the particular version; updating a versioned dataset catalog (216C) to include the at least one structured query and lineage associated with the updated dataset version, the lineage indicating that the updated versioned dataset includes a successor version of the particular version, wherein the versioned dataset catalog includes historical structured queries and lineages applied chronologically for each of the plurality of versions; updating a dependency code (216D) associated with the updated version-controlled dataset, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; A database view of the updated version-controlled dataset is generated by selecting rows of the backing dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

2. The device according to claim 1, characterized in that The backing dataset is implemented as a table comprising rows and columns, wherein the original version of the versioned dataset is stored in its entirety and the subsequent versions of the versioned dataset are stored as incremental operations on the previous version by storing incremental rows and / or columns and by storing indications of operations on the previous version in metadata of the versioned dataset catalog associated with the subsequent version.

3. The device according to any one of the preceding claims, characterized in that The backing dataset includes a single backing dataset storing the multiple versions of the version-controlled dataset.

4. The device according to claim 3, characterized in that Information for compressing the single backup data set is also included, the information including a default value indicating deleted data.

5. The device according to any one of the preceding claims, characterized in that The dependency encoding indicates which rows and / or columns of the updated versioned dataset match rows and / or columns of the specific version, and which rows and / or columns of the updated versioned dataset are deletions relative to the specific version.

6. The device according to claim 5, characterized in that The dependency encoding is implemented as a bitmap, wherein, in response to new rows and / or columns being added to the updated version-controlled dataset, the dependency encoding bitmap includes a Boolean value for each corresponding row, and the Boolean value is set to TRUE if and only if the corresponding row participates in the updated version-controlled dataset, and is set to FALSE otherwise.

7. The device according to any one of the preceding claims, characterized in that The backing data set stores a plurality of default values ​​indicating mappings to values ​​stored in a previous version.

8. The device according to any one of the preceding claims, characterized in that The information refers to code including additional extensions and / or an application programming interface (API) (206B) that interfaces with a relational database management system (RDBMS) (250) that manages the version-controlled dataset.

9. The device according to any one of the preceding claims, characterized in that The metadata stored in the version-controlled dataset directory is shared by and readable by multiple users designated to access the version-controlled dataset, wherein concurrent operations for creating and / or revising the multiple versions of the version-controlled dataset requested by the multiple users on the version-controlled dataset are encapsulated as transactions.

10. The device according to any one of the preceding claims, characterized in that The lineage of the corresponding versions stored in the version-controlled dataset directory represents the dependencies between the multiple versions of the version-controlled dataset and is selected from the group consisting of: the direct ancestors of the corresponding version, one of the multiple ancestors of the previous version, the root ancestor indicating the first version without a previous version, and the direct ancestors of two or more merged versions.

11. The device according to any one of the preceding claims, characterized in that The database view stores a mapping indicating calculations of columns and / or rows of the updated versioned dataset based on columns and / or rows of one or more previous versions, wherein the view definition of the updated versioned dataset defines instructions for applying the calculations to the columns and / or rows of the one or more previous versions based on the mapping.

12. The device according to claim 11, characterized in that The computation includes one or more members selected from the group consisting of: rename, negate, type conversion, add, concatenate, join, any unary and n-ary mappings available in the RDBMS, and user-defined mappings.

13. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to: re-executing the chronologically applied historical structured query stored in the version-controlled dataset catalog to create an updated version of the particular one of the plurality of versions of the version-controlled dataset based on current real-time data; storing the updated version in the version-controlled dataset; The version-controlled dataset catalog is updated with the lineage and an indication of re-execution of the updated version.

14. The device according to any one of the preceding claims, characterized in that The at least one structured query includes instructions to insert new rows and / or new columns into the particular version, the backing dataset is updated to include the new rows and / or new columns, and the dependency encoding is updated to indicate that the new rows and / or new columns are associated with the updated version-controlled dataset, wherein the new rows and / or new columns are not associated with other versions.

15. The device according to claim 14, characterized in that The view of the updated versioned dataset created by selecting rows of the backing dataset corresponding to the updated versioned dataset and projecting columns includes the new rows and / or new columns.

16. The device according to any one of the preceding claims, characterized in that The at least one structured query includes instructions to delete rows and / or columns from the specific version, the version-controlled dataset catalog is updated to indicate that the rows and / or columns of the specific version are excluded from the updated version-controlled dataset, wherein the rows and / or columns are not deleted in the backup dataset, the dependency encoding is updated to indicate that the deleted rows and / or deleted columns are not associated with the updated version-controlled dataset, and the deleted rows and / or deleted columns remain associated with other versions.

17. The device according to claim 16, characterized in that The view of the updated versioned dataset created by selecting rows of the backing dataset corresponding to the updated versioned dataset and projecting columns does not include the deleted rows and / or columns.

18. The device according to any one of the preceding claims, characterized in that The at least one structured query includes instructions to: change values ​​in rows and / or columns of the particular version, copy the rows and / or columns affected by the changed values, update the backing dataset to include the copies, perform the change to the values ​​on the copies, and update the affiliation encoding to indicate that the changed copy with the values ​​is associated with the updated versioned dataset, wherein the changed copy with the values ​​is not associated with other versions.

19. The device according to claim 18, characterized in that Unchanged values ​​in the copy associated with the updated versioned dataset are stored as default values, the default values ​​indicating a mapping to corresponding data stored in the particular version, wherein the view of the updated versioned dataset is rendered by replacing the default values ​​with the corresponding data according to the mapping.

20. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to detect orphaned rows and / or columns in the backing data set that are not associated with any of the plurality of versions and automatically delete the orphaned rows and / or columns from the backing data set.

21. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to: analyzing configuration parameters indicating a tradeoff between storage cost and access cost for one or more of the plurality of versions and the updated versioned dataset; A new backing data set is created for the one or more versions, wherein the configuration parameters indicate that the storage cost is lower than the access cost.

22. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to: Converting the at least one structured query into an abstract syntax tree (AST); manipulating the AST by replacing property references to obtain a manipulated AST; Transforming the operated AST into a rewritten structured query; The rewritten structured query is applied to the backing data set.

23. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to: preparing the view and / or the backing dataset to operate on the updated versioned dataset using rewrite rules for processing at least one structured query, The at least one structured query is applied to a current version of the view and / or the backing dataset to operate on the updated versioned dataset.

24. The device according to any one of the preceding claims, characterized in that Also included is information that, when executed by the processor, causes the processor to: selecting one of the plurality of versions stored in the backup dataset; Verify that the selected version has no dependent versions; applying at least one structured query to the selected version; updating metadata of a versioned dataset catalog associated with the selected version to further include at least one structured query applied to an existing chronologically applied historical structured query; The view of the selected version is updated based on the application of at least one structured query.

25. The device according to any one of the preceding claims, characterized in that The at least one structured query includes instructions to sample and / or partition at least a subset of the specific version; creating a versioned dataset comprising said updated version of said at least one subset; wherein the version-controlled catalog of the updated version-controlled dataset comprises a lineage indicating that the at least one subset of the updated version-controlled dataset is derived from the particular version.

26. A device according to any one of the preceding claims, characterised in that The at least one structured query includes an instruction to randomly arrange the data of the specific version; creating the updated versioned dataset including the randomly permuted data, wherein the randomly permuted data is implemented as at least one of: a mapping of the particular version of the data to a new order of the updated versioned dataset, and a replication of the particular version of the data to the new order of the updated versioned dataset; the versioned catalog of the updated versioned dataset includes instructions indicating that the updated versioned dataset having the randomly permuted data is a lineage derived from the particular version and instructions to randomly permutate the data of the particular version to create the updated versioned dataset; A dependency map stores the new order of the updated versioned data sets.

27. A device according to any one of the preceding claims, characterised in that Also included is information that, when executed by the processor, causes the processor to: receiving an instruction to purge the specific version; Deleting from the backup dataset data that is exclusive to the specific version and not present in any dependent datasets derived from the specific version; deleting from the versioned dataset catalog metadata that is exclusive to the particular version and that is not present in any dependent datasets derived from the particular version; The metadata of the versioned dataset catalog is updated for any dependent datasets derived from the particular version by changing the dependency from the purged version to an earlier version that the purged version depended on.

28. The device according to claim 2, characterized in that Also included is information that, when executed by the processor, causes the processor to: receiving an instruction to merge a first version with a second version, the first version and the second version being derived from a common superior version; creating a merged set of operational operations by integrating operational operations applied to the first version with operational operations applied to the second version; applying the merged set of operations to the common superior version; A set of operations indicating the lineage derived from the first version and the second version and the merge is stored in the metadata of the versioned dataset catalog associated with the updated versioned dataset.

29. A method for managing a version-controlled dataset, characterized in that include: Receiving (104) at least one structured query, the at least one structured query indicating an instruction to operate on a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backing dataset; creating (108) an updated version-controlled dataset in the backing dataset by applying the at least one structured query to the particular version; The updated version-controlled dataset is stored (110) in the backup dataset, wherein The updated versioned dataset stores incremental changes to the specific version without creating redundant copies of the specific version's data; updating a versioned dataset catalog (114) to include the at least one structured query and lineage associated with the updated dataset version, the lineage indicating that the updated versioned dataset includes a successor to the specific version, wherein the versioned dataset catalog includes historical structured queries and lineages applied in chronological order for each of the plurality of versions; updating a dependency code (116) associated with the updated version-controlled dataset, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; A database view of the updated version-controlled dataset is generated by selecting rows of the backing dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

30. A computer program product for a version-controlled dataset, characterized in that A non-transitory medium storing a computer program that, when executed by at least one hardware processor, causes the at least one hardware processor to: receiving at least one structured query indicating an instruction to operate on a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backup dataset; creating an updated version-controlled dataset in the backing dataset by applying the at least one structured query to the specific version; The updated version-controlled dataset is stored in the backup dataset, wherein: the updated versioned dataset stores incremental changes to the specific version without creating redundant copies of the specific version's data; updating a versioned dataset catalog to include the at least one structured query and lineage associated with the updated dataset version, the lineage indicating that the updated versioned dataset includes a successor version of the specific version, wherein the versioned dataset catalog includes historical structured queries and lineages applied in chronological order for each of the plurality of versions; updating a dependency code associated with the updated version-controlled dataset, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; A database view of the updated version-controlled dataset is generated by selecting rows of the backing dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

31. An apparatus (204) for managing a version-controlled dataset (216B), characterized in that include: A processor (202) configured to: Receiving at least one structured query (212A), the at least one structured query indicating an instruction to operate on a specific version of data, the specific version being from a plurality of versions included in the version-controlled dataset, wherein the version-controlled dataset is stored in a backing dataset (216A); creating an updated version-controlled dataset in the backing dataset by applying the at least one structured query to the specific version; storing the updated version-controlled dataset in the backing dataset, wherein the updated version-controlled dataset stores incremental changes to a particular version without generating redundant copies of the data for the particular version; updating a versioned dataset catalog (216C) to include the at least one structured query and lineage associated with the updated dataset version, the lineage indicating that the updated versioned dataset includes a successor version of the particular version, wherein the versioned dataset catalog includes historical structured queries and lineages applied chronologically for each of the plurality of versions; updating a dependency code (216D) associated with the updated version-controlled dataset, the dependency code indicating which data of the updated version-controlled dataset matches the data of the specific version and which data of the updated version-controlled dataset is deleted relative to the data of the specific version; A database view of the updated version-controlled dataset is generated by selecting rows of the backing dataset corresponding to the updated version-controlled dataset according to the dependency code and projecting columns.

Images