Creating a database state tree

Abstract

To provide a computer-implemented method, a program, a storage medium, and a database for creating a tree of database states.SOLUTION: The method comprises: a step S10 of providing a database having at least one branch of database states; a step S20 of receiving, by the database, one or more write events that are applied to an identified state of the database; a step S30 of recovering the identified database state from the database; and a step S40 of creating a new branch by creating a new patch as an alternative state to the identified database state with a reference to the identified state.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more particularly, to a method, system, and program for creating a tree of database states to provide the ability to create, query, and compare alternative states of a database.

Background Art

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, and refers to software solutions for designing objects, for example. CAE stands for Computer-Aided Engineering, and refers to software solutions for simulating the physical behavior of future products, for example. CAM stands for Computer-Aided Manufacturing, and refers to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, graphical user interfaces play a crucial role in terms of technical efficiency. These technologies can be integrated into Product Lifecycle Management (PLM) systems. PLM is a business strategy that helps companies share product data, apply common processing, and leverage enterprise knowledge to help develop products from concept to lifecycle, across the concept of an extended enterprise. Dassault Systèmes' PLM solutions (under the trademarks CATIA, ENOVIA, and DELMIA) provide an Engineering Hub for organizing product engineering knowledge, a Manufacturing Hub for managing manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to the Engineering and Manufacturing Hubs. Together, these systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, facilitating product definition, manufacturing preparation, production, and service optimization.

[0003] These applications are examples of "creative authoring applications," which provide users with the ability to work remotely, explore various solutions, access work in different states, and navigate the history of changes in a "time-travel" manner, thereby solving problems incrementally. "Time travel" means being able to efficiently access past states of a database (i.e., with a controlled elapsed time considered constant for all states) and execute read-only queries against these states. Saving a history of changes and accessing past states of a database in a time-travel manner can also be important in authentication processes, such as the process of auditing and understanding past changes in their context, which is already present in the aviation sector.

[0004] Such creative authoring applications involve long-running transactions, in contrast to short-running transactions. Long-running transactions, also called long-running transactions, are computer database transactions that avoid locking non-local resources, use compensation to handle failures, and typically use a coordinator to complete or interrupt the transaction. Compensation restores the original state or an equivalent state. Therefore, especially when multiple users access the database simultaneously, transactions must respect the so-called ACID properties—indivisibility, consistency, independence, and persistence—to ensure the accuracy and integrity of the data stored in the database even in the event of errors or power failures. In the context of databases, a series of operations on a database that satisfy the ACID properties (these operations can be perceived as a single logical operation on the data) is called a transaction. For example, moving funds from one bank account to another is a single transaction, even if it involves multiple changes such as withdrawals from one account and additions to another.

[0005] Of the ACID properties, consistency must be defined in relation to the meaning (semantics) defined by the application, and therefore cannot be discussed for general databases.

[0006] In particular, for creative authoring applications, it is desirable to be able to open transactions to any state of any branch of the database in order to create, query, and compare alternative states of the database to provide performance that conforms to the workflow of the creative authoring application. This is the concept of version control for datasets. The concept of database version control is similar to version control in software development, but in database version control, instead of tracking software code, it tracks the state of a dataset. Various strategies exist for known database systems to provide one or more of the above performance for a database.

[0007] In "DataHub: Collaborative Data Science & Dataset Version Management at Scale" by Bhardwaj et al. (7th Biennial Conference on Innovative Data Systems Research, 2015), users are provided with the capability to perform collaborative data analysis based on this version control system. To achieve this goal, two main data representations are used: version-first representation and record-first representation. In version-based representation, data corresponding to a specific branch is stored in a dedicated location (i.e., an SQL table), while in record-first representation, data is stored as a list of records, and each list is annotated with the version to which it belongs. [Overview of the project] [Problems that the invention aims to solve]

[0008] A common drawback of these two approaches is that the concept of alternatives, or "versions," exists at the data level. Therefore, each data structure within the database must be aware of this concept, adding to the cost of software development and maintenance. Furthermore, answering queries requires reconstructing the data structure of a specific alternative state (or reusing a materialized state), incurring corresponding resource costs. This resource requirement is further exacerbated when applying these tools to big data or to creative authoring applications that navigate the history of changes in a time-traveling manner.

[0009] Against this backdrop, there is still a need for improved methods to create, query, and compare alternative database states that combine the time-travel characteristics of the authoring system with the performance of long-running transactions, while reducing the memory generated by computing resources and I / O. More generally, there is a need for improved methods for creating database state trees. [Means for solving the problem]

[0010] Accordingly, a computer implementation method for creating a database state tree is provided, the method comprising the steps of: providing a database having at least one branch of the database state; the database receiving one or more write events that apply to the database's identification state; recovering the database's identification state from the database; and creating a new branch by creating a new patch as an alternate state for the database's identification state, along with a reference to the identification state.

[0011] A computer implementation for creating a database state tree may include one or more of the following: - The step to recover the database's identification state is: - Step (S210) to retrieve the branch patch of the identification state in the sequence of database states from the database storage. - Step (S220) to obtain a list of logged write events that occurred between the acquired patch and the identification state. • Includes the step (S230) of applying the list of logged write events to the acquired patch, - The step of creating a new branch by creating a new patch further includes the step of flushing the buffer to database storage if the database state of each alternate state is not a patch stored in database storage. -Reference to identification status: • If the identification status is represented by a patch, refer to the acquired patch, and • If the identification status is not represented by the patch, include references to patches created after the buffered pages were flushed to database storage. - A new patch contains a list of new pages modified or created by a new write event, thereby obtaining a sequence of states where each state has a corresponding patch. - The new patch further includes a mapping table that contains a mapping between the physical address of a buffered page and the logical address of the page in the database's data structure layer. -Each logged write event has a mapping table, which is created by duplicating the existing mapping table and applying the local mapping up to the desired logged write event. - New patches include: -Includes a descriptor, which is: • The number of physical pages modified or created by a new write event. • Metadata to verify descriptor integrity, • Timestamp of new write events • Referencing the corresponding identification status in the database Among them, including at least one of - Database storage is append-only; - This method further includes logging each of one or more write events received by the database onto a log storage before obtaining a list of logged write events, and writing a history of changes as an event log to the log storage for an intermediate transaction that is a transaction before flushing to the database storage. - Log storage is append-only.

[0012] A computer program including instructions for executing this method is further provided.

[0013] Furthermore, a database including a computer-readable storage medium on which the computer program is recorded is provided.

[0014] Furthermore, a computer-readable storage on which the computer program is recorded is provided.

[0015] Furthermore, a computerized system including the database is provided. Next, embodiments of the present invention will be described based on non-limiting examples with reference to the accompanying drawings.

Brief Description of Drawings

[0016] [Figure 1] A flowchart of an example of a method for creating a tree of database states is shown. [Figure 2] A flowchart of an example of a method for recovering an identification state of a database from the database is shown. [Figure 3] A schematic diagram of an example of changing a data page and recording the changed data page in an append-only storage is shown. [Figure 4] An example of creating a new branch is shown. [Figure 5] An example of a system is shown.

Best Mode for Carrying Out the Invention

[0017] Referring to the flowchart of FIG. 1, a computer-implemented method for creating a tree of database states is proposed. This method includes the step (S10) of providing a database having at least one branch of database states.

[0018] A "database" means a collection of data (i.e., information) organized for search and retrieval (e.g., a relational database based on a predetermined structured language such as SQL). When the database is stored in memory, rapid search and retrieval by a computer become possible. In fact, the database is configured to facilitate data storage, retrieval, modification, and deletion in combination with various data processing operations. The database may be composed of a file or a set of files that can be divided into records, and each record is composed of one or more fields. A field is the basic unit of data storage. A field is also called a page. The pages of a database are an internal basic structure for organizing data within a database file. A page is the basic unit of I / O operations. The page size varies depending on the implementation of the database. For example, in an SQL database server, the page size is 8 kB.

[0019] The user may mainly obtain data from the database via a query. The user can use keywords and sorting commands to quickly search, rearrange, group, and select fields of a large number of records, and obtain or create a report on a specific aggregation of data according to the rules of the database management system in use.

[0020] The state of a database is the set of data stored at a particular point in time. A database is always in one specific state. Adding, deleting, or modifying the information stored in the database changes its state. For example, a database may contain information about various designs of a CAD model characterized by parameters such as length and / or shape. As an example, if one of these parameters is changed during the design process and the new set of parameters is saved to the database, its state changes.

[0021] A database can be represented as a conceptual model with several layers or levels of abstraction, each providing a conceptual model or algorithm that can be independent of a particular implementation. For example, a database may include a data structure layer and a storage layer. The storage layer (also called the physical level) is responsible for storing data in persistent memory, such as a hard drive. The storage layer may use both volatile and persistent memory to store data. The data structure layer (also called the logical level) describes how data is stored in the database using data structures. Data structures define how data is modeled and organized for a specific purpose, such as persistent storage, and the operations and data type implementations allowed on the data are most database-specific at this level.

[0022] Each branch of a database can be considered a sequence of database states, and the state of each database within a branch is obtained by modifying another database previously within the same branch. A fork is the act of creating a second branch from a first branch and represents the relationship between the second branch and the first branch. Alternate states are the states of a database accessed on different branches of the database.

[0023] The method further includes the step (S20) in which the database receives one or more write events that apply to the database's identification state. A write event is a request from a system communicatively coupled to the database to write information to the database, for example, a computer running a CAD application. Writing information to the database includes adding new data to the database and / or removing (i.e., deleting) information already stored in the database and / or modifying data already stored in the database. The database may allow multiple users working with the database to send write requests concurrently or substantially concurrently. The database may queue multiple write requests and execute them in order. Thus, a write event is an operation that changes the state of the database after execution by the database, and execution by the database is the commit of a transaction that includes a write event.

[0024] This method further includes the step (S30) of recovering the database's identity state from the database. Recovering the identity state from the database means retrieving, or restoring, the identity state value from the database storage. The identity state can be retrieved directly from the database storage or restored by combining information from various items stored in the database storage. An example of the latter is restoring the database state from stored log files, such as in a logging strategy, where changes are first recorded in log files and written to non-volatile storage. Log files are files that store a chronological record of actions.

[0025] This method further includes the step (S40) of creating a new branch by creating a new patch as an alternate state of the recovered identification state of the database, along with a reference to the identification state. The step of creating a patch means calculating the patch. The patch contains a list of new pages that have been modified or created by one or more new write events. For example, the new patch (the last calculated patch) lists the differences between the new patch and the current patch, i.e., the last patch obtained before the new patch was calculated. A reference is part of the encoded information that identifies the state. The state may be the identification state of an alternate state.

[0026] This method can be used with various types of databases, including graph database systems such as in-memory RDF services. It improves the performance of creating, querying, and comparing different alternate states of a database. This can be understood as Git-like performance to languages ​​such as SQL and SPARQL with time-travel capabilities. The branching performance of this method is applied at the storage layer level using the entire software stack, whereas traditionally it is at the data structure layer level. Therefore, all existing or new data structures can unknowingly possess this branching performance, even if no "versionable" version of that data structure exists. Being a versionable data structure means that the data structure has persistence, meaning that when modified, it retains the previous version itself. This performance is delivered at low CPU and memory costs, and also at low software development and maintenance costs. Because queries and algorithms run identically across all branches, without the concept of a specific branch being executed, this method provides branching performance for creating, querying, and comparing alternate states of a database without affecting the data structure. Storage costs are limited to new changes in each branch, and there is no duplication of data or pages. As a result, the CPU and memory costs for creating a transaction become the same as if there were no branching capabilities.

[0027] This method is performed by computer. That is, the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, possibly fully automatically or semi-automatically. As an example, the initiation of at least some steps of the method may be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the need to implement the user's wishes. As an example, this level may be user-defined and / or predefined.

[0028] Figure 5 shows an example of a system, where the system is a server, for example, a server that hosts a database.

[0029] The server in this example comprises a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The server may further comprise a video random access memory 1100 and associated graphics processing unit (GPU) 1110 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices like EPROMs, EEPROMs, and flash memory devices, magnetic disks like internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the above may be complemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). One or more mass storage devices may be used to implement the database storage layer. A network adapter 1050 manages access to the network 1060. The server may also include haptic devices 1090 such as a cursor control device and a keyboard. The cursor control device is used by the server to allow the user to selectively position the cursor at any desired position on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generators for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the mouse buttons are used to generate signals. Alternatively or additionally, the server system may include a pressure-sensitive pad and / or a pressure-sensitive screen.

[0030] The computer program may include instructions that can be executed by a computer, and the instructions include means for causing the system to perform the Method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented, for example, as a device such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the Method may be performed by a programmable processor that executes a program of instructions to perform the functions of the Method by acting on input data and producing output. Thus, the processor may be programmable or coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. The application program may be implemented in a high-level procedural programming language or an object-oriented programming language, or in assembly language or machine language, as necessary. In any case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In any case, the application of the program on the system results in instructions for performing the Method.

[0031] A typical example of a computer implementing this method is running it on a system adapted for this purpose, such as a server. The system may include a memory-coupled processor, where memory stores computer programs containing instructions for performing this method. Memory may also store a database. The memory is hardware adapted for such storage and may include several physically distinct components (e.g., one for the program and possibly one for the database).

[0032] In one example, this method is used in the process of generally manipulating modeled objects using an authoring application. A modeled object is any object defined by data stored, for example, in a database. Therefore, the expression "modeled object" refers to the data itself. Depending on the type of system, modeled objects may be defined by various types of data. The system can actually be any combination of CAD systems, CAE systems, CAM systems, PDM systems, and / or PLM systems. In these various systems, modeled objects are defined by corresponding data. Thus, CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, and CAE data can be discussed. However, since modeled objects can be defined by data corresponding to any combination of these systems, these systems are not exclusive to each other. Therefore, the system may be both a CAD system and a PLM system.

[0033] For the sake of clarity, we will discuss examples of methods used in CAD systems.

[0034] The term CAD system further refers to a system, such as CATIA, that is adapted to design modeled objects based on their graphical representation. In this case, the data defining the modeled object includes data that enables the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, and in certain cases faces or surfaces. Lines, edges, or surfaces may be represented in various ways, with non-uniform rational B-splines (NURBS) being an example. Specifically, a CAD file may contain specifications from which geometry may be generated, thereby enabling the generation of what is represented. The specifications of a modeled object may be stored in a single CAD file or multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. Also, a modeled object can typically be an assembly of thousands of parts.

[0035] In the context of CAD, modeled objects are typically 3D modeled objects, which represent products such as parts or assemblies of parts, or in some cases, product assemblies. A "3D modeled object" means an object that is modeled with data that enables a 3D representation. 3D representation allows parts to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, may be processed and rotated around any axis of its or any axis of the displayed screen. This excludes 2D icons, in particular, that are not modeled in 3D. Displaying 3D representations makes design easier (i.e., it statistically improves the speed at which designers perform tasks). Since product design is part of the manufacturing process, this speeds up the manufacturing process in industry.

[0036] The 3D modeled object may represent the geometry of a product after its virtual design has been completed, for example, by a CAD software solution or CAD system. The product may be a (e.g., mechanical) part or assembly of parts (an assembly of parts may be considered a part itself from the perspective of this method, or parts and assemblies of parts are equivalent, as this method can be applied independently to each part of an assembly), or more generally, an assembly of any rigid body (e.g., a movable mechanism). CAD software solutions enable the design of products in a wide range of industries without limitation, including aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. The 3D modeled objects designed by this method may be industrial products that are any mechanical parts, such as parts for land vehicles (e.g., automobile and light truck equipment, racing cars, motorcycles, truck and motor equipment, trucks and buses, trains, etc.), parts for aircraft vehicles (e.g., airframe equipment, aerospace equipment, propulsion equipment, defense products, aircraft equipment, space equipment, etc.), parts for marine vehicles (e.g., naval equipment, commercial ships, offshore equipment, yachts and workboats, marine equipment, etc.), general mechanical parts (e.g., industrial manufacturing machinery, heavy machinery or equipment, installation equipment, industrial equipment products, metalwork products, tire manufacturing products, etc.), electrical machinery or electronic components (e.g., home appliances, security and / or control and / or measurement products, computing and communication equipment, semiconductors, medical devices and equipment, etc.), consumer goods (e.g., furniture, home and garden products, leisure goods, fashion products, products of durable goods retailers, products of textile retailers, etc.), and packaging (e.g., food and beverages and tobacco, beauty and personal care, household goods packaging, etc.).

[0037] The CAD design process is typically collaborative, requiring multiple individuals to work independently or collaboratively. In this context, it is crucial that the CAD system's database provides all users with independence and integrity of the database state. Such design applications require creative authoring that involves exploring various design possibilities incrementally and navigating the history of changes in a time-traveling manner, as well as the efficient creation, querying, and comparison of alternative database states. Examples of the methods of this invention are described against the backdrop of a CAD authoring application. It should be understood that these examples are not limited to CAD authoring applications but can be applied to any authoring application. More generally, these examples of the methods are not limited to a specific area of ​​authoring applications; that is, examples of the methods can be applied to any database that stores database states.

[0038] Refer again to Figure 1 for further explanation. Step S10 provides a database having at least one database state branch. As an example, the database may have multiple database state branches. For example, this database may contain information on a CAD model design collected by one designer or multiple designers working in parallel on the CAD model through an incremental creative application. As an example, each branch may be attributable to a different group of tasks in the design process and may contain collected information on a CAD model design provided by a group of designers in the design process. It should be understood that the database may include a set of data pages as the smallest unit, as mentioned above, but any other basic internal structure may be used to organize the data within the database.

[0039] For example, database storage stores records in pages, and database storage is append-only. Therefore, the database is immutable, and the entire history of all transactions is stored in event log storage. This is useful for auditing and historical queries.

[0040] Next, in step S20, the database receives one or more write events that apply to the database's identification state. A write event is a user request to write information to the database, adding one or more new data or modifying one or more existing data already stored in the database. Multiple write transactions may send multiple write events simultaneously by multiple users. Each write event contains a set of information to update the state of the database. Write events may be sent via an application programming interface (API) or via a remote procedure call (RPC), which typically uses pipes, sockets, or shared memory. For example, if a group of designers working on the same product simultaneously modify 3D parts of an assembly of components that make up the product, each modification of the component performed by one or more designers (i.e., the received write event) generates the set of information that applies to update the database.

[0041] Next, in step S30, the database's identity state is recovered from the database. Recovering the identity state from the database means retrieving its value from the database. This identity state may be queried by a user or automatically by a computer. The recovery of the identity state may vary depending on how the database state is stored in database storage. A preferred example of storing the database state in database storage is described below.

[0042] A preferred example of saving database state to database storage is that write events received by the database are logged, and each logged write event forms a new state in the database; that is, the commit of a write transaction containing a write event forms a new state in the database. The step of logging write events is the operation of keeping the log in non-volatile memory. It is understood that any type of non-volatile memory (also called long-term persistent storage) may be used. A log file is a file that stores a chronological record of operations performed by the database in response to requests from a system that is communicatively coupled to the database. Therefore, logging write events means saving the write events received by the database at time t in the log file, after the immediately preceding write event received at time t-1. Thus, the log file contains the set of write events received by the database. The log file contains the changes (write events) applied to the saved data in the process of committing a write transaction, where the write event is within a write transaction. A write transaction is a unit of read / write events in a database that is performed while maintaining data integrity. The transaction itself consists of a set of information for updating the state of the database. The actual state of the target database can be reconstructed from the known state by applying logged changes (i.e., write events). Write events change the state of the database from the perspective of the write transaction.

[0043] This preferred example of storing the state of a database in database storage further includes the step of buffering pages that have been modified or created by one or more write events. As mentioned earlier, a database page is the internal basic structure for organizing data within a database file, and a page is the basic unit of I / O operations. A database page is a contiguous block of database memory as the smallest unit of data for database management. The database architecture allocates all existing or new data structures and stores specific information about the data structures within the pages. For example, a page has a physical identification in the storage hardware (where the page physically resides) and a logical identification used to ensure that the page has a fixed identifier even if the page changes over time and its physical identification differs. For example, the storage layer is the file system allocator, which functions as the data structure layer. Such a mapping between logical and physical pages separates the data structures from the storage layer's concurrency control model. Conceptually, it is necessary to have a consistent mapping for all pages of transactions performed on a particular identifier. In some examples, the identifier may be a timestamp, which is information encoded in a string that identifies when a particular event occurred. In other examples, the identifier may be a flag indicating the state of the database (e.g., "version 1.0"). The buffering step involves temporarily storing data in a specific area of ​​physical memory called a buffer. Hereafter, this buffer will be referred to as the shared change state. These two terms may be used interchangeably and with the same meaning. The shared change state is the state of modified pages shared between transactions. Therefore, the shared change state can originate from a transaction, persist without a transaction, and combine with another shared change state to create a new shared change state, which conceptually corresponds to the joining of two transactions. Thus, the shared change state may be linked to a transaction until the transaction is successfully committed or aborted.If a transaction is interrupted, the shared change state may be discarded. Once the first transaction is committed, its shared change state is no longer linked to the first transaction and may become available to subsequent transactions. This shared change state then becomes considered all modified pages since the preceding patch (i.e., its meaning has changed, and it is no longer a "page modified by a specific transaction," but rather a "page modified since the preceding patch").

[0044] A preferred example further includes the step of creating a patch by flushing buffered pages to database storage when a threshold is reached. The step of creating a patch means calculating the patch. The patch contains a list of new pages modified or created by one or more new write events. For example, the new patch (the last calculated patch) lists the differences between the new patch and the current patch, i.e., the last patch obtained before the new patch was calculated. The act of flushing means emptying the shared change state and writing its contents to non-volatile memory. The threshold for this flush may be, for example, the size of the log and / or the size of buffered pages and / or the time to rebuild from the log and / or the time elapsed since the last flush was performed. The time to rebuild from the log is the estimated time it takes to reapply the events recorded in the event log storage to rebuild the identification state.

[0045] In a preferred example, each write event received by the database is logged. For example, this logging is performed by maintaining one or more files (also called log files) that store a history of the changes made to the stored data during the process of at least one write event. In other examples, logging may be performed using a shared log queue or a distributed file system. The log stores a history of all changes made by one or more consecutive write events, including all write events received by the database. Each logged write event creates a new state in the database, and therefore the log file stores information that allows for the recovery of the new state; in other words, the log stores the database transactions made by the received write events. It should be understood that the committed state is accessible in a time-travel manner. Changes to the state that have not yet been committed are made by write operations that create log events in memory and new pages in memory. At commit, the log events are written to disk, and the pages are buffered or written to disk.

[0046] For example, the step of logging each write event includes writing the history of changes made by the received write event as an event log (in the form of an event log) to event log storage on memory and / or temporary storage, such as temporary disk space or a temporary file, for intermediate transactions, which are transactions before they are flushed to database storage (e.g., on disk). An intermediate transaction (also called an unflushed transaction) is a transaction that has been triggered by the received write event and has been saved in the log, but has not yet been associated with the persistent state of the database. In this situation, logging acts as a complete collection of all transactions made by the received write event, with one or more transactions that have not yet been flushed to the database storage of the database saved as intermediate transactions.

[0047] For example, event log storage is append-only, meaning new logs are always appended to the file rather than replacing existing data. Append-only log storage records data changes that occur by writing each change to the end of the file. In this case, the entire set of received write events can be recovered by replaying the append-only log from beginning to end. For example, one or more past write events (whether flushed to data storage or not) can be replayed by replaying one or more transactions and / or one or more intermediate transactions stored in the append-only event log storage.

[0048] In a preferred example, pages modified or created by an received write event are buffered. For instance, these modified or created pages are the result of a user interaction with a product by a group of designers, and the changes made to the product by the designer group are saved. Buffering means that pages modified or created by an received write event are stored in memory so that the system can process these pages at a later stage. Thus, modified versions of pages from a write event are queued in a memory write buffer and flushed at a later stage. These modified versions of pages constitute a set of pages called the shared modified state. It should be understood that database pages are selected for illustrative purposes only, and other database fundamental units of I / O operations may be used.

[0049] These pages, modified or created by the received write events, remain in buffer memory until a threshold is reached, at which point the buffer memory is flushed (S50). The creation of patches resulting from the flushing of buffer memory is described below.

[0050] For example, the threshold may be the size of the event log. Therefore, if the event log size exceeds or equals the threshold, the buffer memory is flushed. The event log may be a log file, as already described. The event log size may include the number of received write events stored in the event log, and / or the space occupied by the event log on the storage medium where the event log is stored (e.g., measured in megabytes (MB) or gigabytes (GB)). For example, the threshold may be set to 16MB, in which case, buffered pages are flushed when the event log size exceeds 16MB.

[0051] For example, the threshold may be the size of the buffered pages. Therefore, if the size of the buffered pages exceeds or equals the threshold, the buffer memory is flushed. The size of the buffered pages may include the number of pages stored in the buffer and / or the space occupied by the buffered pages on the buffer (e.g., measured in megabytes (MB) or gigabytes (GB)). For example, the threshold may be set to 1GB, in which case the buffered pages are flushed when the number of buffered and stored pages exceeds 1GB.

[0052] For example, the threshold may include the time elapsed since the last flush. Thus, memory flushing is performed periodically, for example, every n seconds, n minutes, n hours, n days, n weeks, or n months.

[0053] For example, the threshold may include the time required to reconstruct from the log. Reconstruction time is an estimate of the time required to replay some or all of the logged events, and therefore reconstruction time is the time to recalculate the set of transactions that occurred or were logged. In this case, the entire set of received write events can be recovered by replaying the append-only log from beginning to end. For example, one or more past write events (whether flushed to data storage or not) are replayed by replaying one or more transactions and / or one or more intermediate transactions stored in the append-only event log storage. The reconstruction time from the last flush may be estimated based on the number of log events in the event log storage since the last flush was performed.

[0054] As described above, a preferred example further involves creating patches for pages that have been stored in a buffer and modified or created by a write event, after these pages have been flushed to database storage. The created patches are sometimes called new patches because the system managing the database may include at least one existing patch obtained previously as a result of flushing the buffered pages. Each created patch may be identifiable by being assigned an identifier (such as a timestamp or an ascending integer index), so that all created patches can be arranged in the order of creation to form a sequence of patches. In this way, the flushed patches are in chronological order of flushing by being assigned their respective identifiers. Therefore, the new patch is the most recent patch created and stored at the end of the file. The new patch contains a list of one or more pages modified or created by the last write event received. The new patch lists only the page changes between the new patch and the second-to-last patch obtained as a result of the second-to-last flush of buffered pages. This improves memory usage and I / O costs.

[0055] Therefore, the creation of a new patch is performed each time the database receives one or more write events and reaches a threshold. With each repetition of creation, a list of new pages modified or created by the new write events received by the database is created. Thus, a sequence of states is obtained in which some states each have a corresponding patch. In fact, there is no guarantee that a patch represents a state, as at least the creation of a patch relies on flushing buffered pages to the database, triggered by reaching a threshold. Therefore, a patch can represent a state in the database, or a combination of one or more states in the database, and / or a part of one or more states in the database. States are recursively made up of patches. The set of pages Pi, which is reconstructed from the set of pages Di of patch i, is recursively reconstructed. This can be written as follows:

number

[0056] Therefore, a preferred example is a hybrid choice for storage methods, using a page version mapping table combined with a memory write buffer on the one hand, and event log storage on the other.

[0057] The conceptual model of a database has already been explained above. As an example, a database includes a storage layer, in which memory can be allocated, modified, and the changes saved. Steps S30 to S50 may be implemented at the storage level. The database further includes a data structure layer, in which data structures (such as B-trees) are created by allocating memory in the storage layer and modifying that memory area to create the necessary structure. Steps S10 to S20 may be implemented at the data structure level. Figure 4 shows an example of the relationship between the data structure layer and the storage layer. Figure 4 will be explained below.

[0058] Refer to Figure 3 for a schematic example of creating a new patch. This example starts with three data pages stored in an additional-only data storage, which have already been flushed in the database. Similar to traditional shadow paging architectures, when n pages are flushed to disk, an n-page patch is created. Each of these pages has a physical identification of the data within the storage layer where each page physically resides; for example, each page has a physical address on the storage medium, such as a disk drive. For example, the database refers to the data pages by a logical identification that is not affected by changes in physical identification; for example, each page has a logical address from the database's perspective. These three pages are "referenced" by the existing patch because they have already been flushed. For this purpose, the patch may include a mapping table. The mapping table must be predefined to establish the mapping between the logical and physical identifications described above. It should be understood that techniques other than using the mapping table may be used to access the files on disk. The patch created after the flushing of the three pages includes a mapping table. The mapping table can be considered a theoretical multi-version index. While a mapping table performs the same function as a theoretical multi-version index, it does not actually implement a full multi-version index for performance improvements. As shown in Figure 3, for timestamp T1, the mapping table maps logical addresses (1, 2, 3) to physical addresses in storage (adr#1, adr#2, adr#3), respectively. For timestamp T2, pages 1 and 3 are modified. The logical addresses remain unchanged; instead, the new mapping table points to the old physical address of page 2 at T1 and the new physical addresses of pages 2 and 3 at T2. When a transaction is created (or opened) in the database, this mapping table is constructed or instantiated in memory between the logical and physical pages. Thus, the data structure recognizes only logical pages without knowledge of the physical pages, and the data structure recognizes only logical pages without knowledge of the physical pages.If a mapping table already exists in memory, the new mapping table is created by duplicating the existing mapping table, for example, the mapping table for the previous timestamp, and applying the local mapping up to the desired transaction and database state. If a mapping table does not exist, all patches are scanned by going back from descriptor to descriptor, for example, from the descriptor of the patch with timestamp T2 back to the descriptor of the patch with timestamp T1, and the latest version of all pages defined by the first descriptor selected for use is obtained. This mapping table strategy reduces the memory cost of a single multi-version index and its high lock contention. The implementation of the mapping table may choose to obtain the fastest table for associations between integers and integers / pointers. As an example, the mapping table may be created in a performance-efficient manner based on a lock-free compare-and-swap array, as is well known in the art. In another example, the mapping table may be created based on a user-space read-copy-update (RCU) synchronization mechanism.

[0059] In the example in Figure 3, a patch includes a list of one or more new pages modified or created by a new write event, and also includes descriptors. A patch representing a set of pages may be subject to several placement constraints to make mapped memory available. For example, 4k bytes per page are allocated to virtual memory on Intel® and Arm® processors, and as another example, the Windows® virtual memory manager may require patches to be placed in 64k-byte blocks, and therefore patches are multiples of 64kb blocks, with descriptors written to the end of the last 64kb block. Descriptors can be added as the last element written to the end of the memory block containing the patch's pages to support indivisibility: the presence of a descriptor indicates that the pages were successfully committed, or in other words, writing this descriptor last provides a simple way to verify that the entire patch is correctly registered. Otherwise, any data found after the previous descriptor is ignored, i.e., discarded. As an example, a descriptor may include at least one of the following: - The number of physical pages modified or created by a new write event. This allows us to find the beginning of the current patch and easily find the previous patch to pre-allocate the memory resources associated with that patch. - Metadata used to check the integrity of the descriptor. This verifies that the descriptor is not corrupted. - Timestamps of write events represented by patches. The sequence of patch commits (i.e., their flushes) can be obtained from the descriptor. This improves database reconstruction when necessary. - A reference to the corresponding identification state in the database. Using this reference, the database can find out which branch the alternate state refers to.

[0060] The presence of descriptors may provide a reliable termination condition for the page mapping algorithm. Referring again to the example in Figure 3, the patch includes a list of one or more new pages modified or created by the last received write event and a descriptor. The patch may further include a mapping table, which includes the mapping between the logical and physical identities.

[0061] The actual state of the target database, particularly the state that does not correspond to the patch in the database storage, can be reconstructed from the known state, i.e., the state with the corresponding patch in the database storage, in combination with the application of changes recorded in the event log files recorded in the event log storage.

[0062] This section describes how to recover the database's identification state from the database. Figure 2 shows an example of recovering the database's identification state from the database. This identification state may be the database state that already exists on disk as a result of flushing to a buffer, or it may be the state that exists chronologically between two saved patches. In either case, in step S210, the method retrieves a patch of the branch of the identification state from the database storage in sequence of states. As previously described, the patch retrieved to construct the database state includes the retrieval of preceding patches, as a patch is generally not sufficient on its own. In the former case, this retrieved patch is the corresponding patch of the database state flushed to disk as a result of a write event. In the latter case, the method retrieves a list of logged write events that occurred between the retrieved patch and the identification state. The patch retrieved in step 210 is the patch associated with the same branch as the identification state. In one example, the retrieved patch is the first patch that comes before the identification state chronologically. In another example, the retrieved patch is the first patch that comes after the identification state chronologically. In yet another example, the retrieved patch is the patch closest to the identification state in chronological order, that is, the patch that minimizes the length of the list of logged write events that occurred between the retrieved patch and the database's identification state. Next, in step S220, the method retrieves a list of logged write events that occurred between the retrieved patch and the database's identification state. These log files hold a history of write events that introduced changes to data pages. In step S230, the identified database is recovered by applying the list of logged write events to the retrieved patch.

[0063] Returning to Figure 1, in the next step S40, a new branch is created by creating a new patch as an alternative state to the identification state, along with a reference to the identification state. If the identification state does not exist on disk as a result of a buffer flush and is a state between two saved patches chronologically, creating a new patch further involves flushing the buffer to database storage. By flushing the state of each new branch to the database, the method can use log files to reconstruct the intermediate states corresponding to each branch.

[0064] Identification state references are added to each newly created patch in a new branch so that the database can discover which branch the alternate state refers to. Finding this corresponding branch is crucial for finding queries about the database state of the new branch. Identification state references include references to the retrieved patch if the identification state is represented by a patch, and references to the patch created after the buffered page was flushed to database storage if the identification state is not represented by a patch.

[0065] Figure 4 shows an example of creating a database state tree as an alternative state according to this method, where the database state stores a data tree. The information in the data structure layer tree is stored in some data pages of the storage layer. As shown in Figure 4, at timestamp T1, the mapping table maps logical addresses (1, 2, 3) to physical addresses in storage (adr#1, adr#2, adr#3), respectively. At timestamp T2, the write event modifies pages 1 and 3, and a corresponding new patch is created in database storage containing only the modified pages, i.e., pages 1 and 3. As a result of this change, the mapping table is updated to map logical addresses (1, 2, 3) to physical addresses in database storage (adr#1, adr#2, adr#2), respectively. The addresses in the patch are local to the patch, not global addresses. The corresponding patches at timestamps T1 and T2 are considered to be on the same branch named B1. At timestamp T4, a fork is applied to the database history at timestamp T1, and a new branch named B2 is created from the database state corresponding to timestamp T1. In this new branch, page 3 is modified compared to the identification state at timestamp T1, and a corresponding new patch is created in database storage along with the modified page 3. In some examples, the creation of the new branch may occur at timestamp T3, and the modification of page 3 may occur at timestamp T4. In this scenario, since branch B2, created at timestamp T3, points to an existing patch, the patch is created only at timestamp T4. The mapping table for the corresponding new patch is updated, mapping logical addresses (1, 2, 3) to physical addresses (adr#1, adr#2, adr#1). As mentioned above, the addresses within the patch are local to the patch, not global addresses. The reference to the patch at timestamp T2 in the branch may be written to the descriptor of the patch at timestamp T4 in branch B2.

[0066] In this invention, a new branch of the database is created without affecting the data structure by manipulating only the logical identifier of the page. Because this strategy is implemented at the database storage level rather than the data structure level, the branching performance of this method is applicable to all data structures in the database. Queries and algorithms executed on these data structures run independently of the branch in which they reside, and are therefore available without the cost of software development and maintenance.

[0067] Furthermore, the cost of the required resources is reduced because there is no duplication of data or pages between the two branches. In this method, opening a transaction in a particular branch creates the corresponding mapping table in the same way as creating a mapping table in a single branch. The difference is that it is necessary to know when to fork and jump to another storage item, which is provided via an identification state reference. The reference may be included in the patch descriptor. Therefore, the CPU and memory costs are the same whether there is one branch or multiple branches. The overhead and storage costs are limited to the storage cost of the pages modified in the new branch.

[0068] Although preferred embodiments of the present invention have been described above, it is understood that various modifications can be made without departing from the spirit and scope of the invention. Therefore, other implementations are also within the scope of the following claims.

Claims

1. A computer implementation method for creating a state tree of a database, Step (S10) of providing the initial state of the database in an identifiable manner, wherein the database has at least one branch of state, the at least one branch being a sequence of database states, and the state of each database in the branch is obtained by modifying the state of another database in the same branch at an earlier time; Step (S20) the database receives one or more write events that apply to the identification state of the database, wherein the identification state of the database is the state of the database that already exists on disk as a result of flushing the buffer. The steps include: holding the pages modified or created by the received write event in a buffer, and constructing a mapping table in the buffer that associates the physical identification information of each held page with the logical identification information of the page in the data structure layer of the database; Step (S30) of recovering the database identification state from the database by obtaining a patch on the branch of the identification state in the sequence of the database state from the database storage, and applying a write event to the obtained patch, starting from the provided database (S210), Step (S40) of creating a new branch by referencing the aforementioned identification state and creating a new patch as an alternative state for the identification state of the database, wherein the created new patch is (a) A list of new pages that have been modified or created by one or more write events, (b) The mapping table flushed from the buffer, and (c) Time-series information showing the patch's timeline A data structure that includes and lists only the page changes between the preceding patch, and A computer implementation method having

2. The step (S30) of restoring the identification state of the database is as follows: Step (S210) of obtaining the patch of the branch of the identification state in the sequence of the database state from the database storage, Step (S220) of obtaining a list of logged write events that occurred between the acquired patch and the identification state from a log file stored in non-volatile memory, Starting from the provided database, the process involves committing a write transaction to apply the list of write events logged after the time-series information corresponding to the acquired patch to the acquired patch, using the mapping table contained in the acquired patch (S230). including The computer implementation method according to claim 1.

3. The step of creating the new branch by creating the new patch further includes, if the state of the database in each of the alternate states is not a patch stored in the database storage, the step of flushing the buffer that stores the pages modified or created by the one or more write events to the database storage. The computer implementation method according to claim 2.

4. The reference to the aforementioned identification state is, If the aforementioned identification status is represented by a patch, the acquired patch is referenced. If the aforementioned identification status is not represented by a patch, refer to a patch created after the buffered page has been flushed to the database storage. The computer implementation method according to claim 3.

5. The step of creating a new branch (S40) is repeated, and each created patch is recursively combined with the previous patch to obtain a sequence of database states. The computer implementation method according to claim 3 or 4.

6. The mapping table is, This includes a mapping between the physical address of a page held in the buffer before flushing and the logical address of a page in the data structure layer of the database, and Used to access files on disk The computer implementation method according to claim 5.

7. The logged write event has a local mapping table obtained by replicating the existing mapping table constructed in the buffer, updating the mapping up to the desired logged write event, and applying the local mapping. The computer implementation method according to claim 6.

8. The aforementioned new patch further includes a descriptor, which is a data element written to the end of the patch, for obtaining the latest version of the page if the mapping table does not exist in memory. The descriptor is, - The number of physical pages modified or created by a new write event. Metadata to verify descriptor integrity, • Timestamp of new write events, - Referencing the corresponding identification status in the database It includes at least one of the following: This is used to verify the integrity of whether the patch was written completely to the end, by detecting that the descriptor was written at the end of the patch and by checking for corruption of the descriptor itself using the metadata. The computer implementation method according to any one of claims 3 to 7.

9. The aforementioned database storage is for appending only. The computer implementation method according to any one of claims 2 to 8.

10. The steps further include logging one or more write events received by the database onto log storage before obtaining the list of logged write events, and saving a history of changes as an event log to the log storage for intermediate transactions, which are transactions before flushing to the database storage. A computer implementation method according to claim 2, or any one of claims 3 to 9 that references claim 2.

11. The aforementioned log storage is for appending only. The computer implementation method according to claim 10.

12. A computer program including instructions for performing the computer implementation method described in any one of claims 1 to 11.

13. A computer-readable storage medium recording the computer program described in claim 12.

14. A database and A computer-readable storage medium recording the computer program described in claim 12, and a computer that reads the computer program from the computer-readable storage medium and executes it against the database, A computer system, including a computer system.

Images