Hybrid shadow paging for saving databases

Abstract

To provide a method, system, and program for hybrid shadow paging for storing a database state.SOLUTION: The method comprises the steps of: providing a database; receiving one or more write events by the database; logging each of the write events resulting in forming a new state on the database; buffering pages modified or created by the write events; and creating a patch by flushing the buffered pages to a database storage if a threshold is reached.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more specifically, to a method, system, and program for hybrid shadow paging for storing databases.

Background Art

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, and refers to software solutions for designing objects, for example. CAE stands for Computer-Aided Engineering, and refers to software solutions for simulating the physical behavior of future products, for example. CAM stands for Computer-Aided Manufacturing, and refers to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, graphical user interfaces play a crucial role in terms of technical efficiency. These technologies can be integrated into Product Lifecycle Management (PLM) systems. PLM is a business strategy that helps companies share product data, apply common processing, and leverage enterprise knowledge to help develop products from concept to lifecycle, across the concept of an extended enterprise. Dassault Systèmes' PLM solutions (under the trademarks CATIA, ENOVIA, and DELMIA) provide an Engineering Hub for organizing product engineering knowledge, a Manufacturing Hub for managing manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to the Engineering and Manufacturing Hubs. Together, these systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, facilitating product definition, manufacturing preparation, production, and service optimization.

[0003] These applications are examples of "creative authoring applications," which provide users with the ability to work remotely, explore various solutions, access work in different states, and navigate the history of changes in a "time-travel" manner, thereby solving problems incrementally. "Time travel" means being able to efficiently access past states of a database (i.e., with a controlled elapsed time considered constant for all states) and execute read-only queries against these states. Saving a history of changes and accessing past states of a database in a time-travel manner can also be important in authentication processes, such as the process of auditing and understanding past changes in their context, which is already present in the aviation sector.

[0004] Such creative authoring applications involve long-running transactions, in contrast to short-running transactions. Long-running transactions, also called long-running transactions, are computer database transactions that avoid locking non-local resources, use compensation to handle failures, and typically use a coordinator to complete or interrupt the transaction. Compensation restores the original state or an equivalent state. Therefore, especially when multiple users access the database simultaneously, transactions must respect the so-called ACID properties—indivisibility, consistency, independence, and persistence—to ensure the accuracy and integrity of the data stored in the database even in the event of errors or power failures. In the context of databases, a series of operations on a database that satisfy the ACID properties (these operations can be perceived as a single logical operation on the data) is called a transaction. For example, moving funds from one bank account to another is a single transaction, even if it involves multiple changes such as withdrawals from one account and additions to another.

[0005] Of the ACID properties, consistency must be defined in relation to the meaning (semantics) defined by the application, and therefore cannot be discussed for general databases.

[0006] In particular, for creative authoring applications, it is desirable to provide optimal performance for mixed read and write workloads, and transactions must be able to be created against any state of the database to provide user activity in an incremental manner, typically allowing users to repeatedly test and validate various scenarios through multiple trials. Furthermore, to comply with creative authoring applications, all states of the database must be accessible efficiently and at the same cost; in other words, all database states are considered equivalent.

[0007] Various strategies have traditionally been employed in database systems to provide one or more ACID properties in a database.

[0008] The initial strategy is shadow paging, a strategy introduced to support long-running transactions such as those in CAD applications, while providing transactional indivisibility, consistency, and persistence at the file level using an additional dedicated storage system. In shadow paging, two versions of a file are maintained: a shadow version and a current version. Only the current version of the file is modified; the shadow version remains unchanged. The shadow version is used to provide indivisibility, consistency, and persistence.

[0009] The drawback of shadow paging is the amount of I / O resources used, because it requires waiting for all pages to be written to disk before committing. Furthermore, database libraries released with shadow paging are limited to two versions, which is quite insufficient for authoring applications, as discussed in the literature "Gray et al., The recovery manager of the System R database manager, ACM Computing Surveys (CSUR), 1981, Vol. 13, No. 2, pages 223-242" and "Chu, MDB: A memory-mapped database and backend for OpenLDAP, Proceedings of the 3rd International Conference on LDAP, Heidelberg, Germany, 2011, page 35".

[0010] The second strategy is a log-first write (WAL) logging strategy to maintain indivisibility and persistence. In this method, changes are first recorded in the log and written to non-volatile storage before being written to the database. The advantage of this strategy is the efficiency of I / O costs. However, for long-running transactions in authoring systems, keeping a history of changes as a log means that reconstructing the desired database state will incur enormous CPU costs. [Disclosure of the Invention] [Problems that the invention aims to solve]

[0011] Against this backdrop, there is still a need for improved methods to obtain a database that combines the time-travel characteristics of the authoring system with the performance of long-running transactions, while reducing the memory generated by computing resources and I / O. [Means for solving the problem]

[0012] Therefore, a computer implementation method for saving the state of a database is provided. The method includes the step of providing a database. The method further includes the step of receiving one or more write events by the database, logging each write event, and each logged write event forming a new state in the database. The method further includes the step of holding pages modified or created by the write events in a buffer, and creating a patch by flushing the pages held in the buffer to database storage when a threshold is reached.

[0013] Computer implementations for storing databases may include one or more of the following: -This method further includes: The process repeats each time the threshold is reached after the database receives a new write event. Each iteration of the process includes creating a list of new pages modified or created by the new write event received by the database, thereby obtaining a sequence of states where some states each have a corresponding patch. -Each newly created patch further includes a mapping table that contains a mapping between the physical address of the page held in the buffer and the logical address of the page in the database's data structure layer. -Each logged write event has a local mapping table, which is created by duplicating the existing mapping table in the buffer and applying the local mapping up to the desired logged write event. - The mapping table is based on a lock-free compare-and-swap array. -The created patch further includes a descriptor containing at least one of the following: • The number of physical pages modified or created by a new write event. • Metadata to verify descriptor integrity, • Timestamp of a new write event, - If a descriptor exists, the logged write event is successfully committed to the memory buffer; otherwise, the logged write event is discarded. - The threshold is the size of the event log and / or the size of the pages held in the buffer and / or the time taken to rebuild from the log and / or the time elapsed since the last flush. -Data page database storage is for appending only. - The step of logging each write event includes, for intermediate transactions which are transactions before flushing to database storage, the step of writing the history of changes to log storage on disk as an event log, and / or - Log storage is for additions only.

[0014] Furthermore, a computer implementation method for accessing the state of a database is provided. This method includes the step of providing a database having at least two database states stored according to a method for storing the state of a database. The method for accessing the state of a database further includes the steps of receiving read events for an identification state by the database, obtaining a patch of a sequence of states from database storage, obtaining a list of logged write events that occurred between the obtained patch and the identification state, and applying the list of logged write events to the obtained patch to recover the identification state of the database.

[0015] In a computer implementation for accessing the state of a database, the acquired patch may be a patch that minimizes the length or size of the list of logged write events that occurred between the acquired patch and the database's identification state.

[0016] A computer program is further provided that includes instructions for performing a method of saving the state of a database and / or a method of accessing the state of a database.

[0017] Furthermore, a database is provided that includes a computer-readable storage medium having the computer program recorded thereon.

[0018] The database may include a transaction manager that links a data structure to a transaction in the storage layer and orchestrates commits, where: - The transaction manager in the data structure layer sends an event log to the storage layer, - The persistence of the event log on the log storage is approved by the storage layer, - After approval, the transaction manager closes the transaction in the storage layer and sends the modified page to the memory buffer.

Brief Description of the Drawings

[0019] Next, embodiments of the present invention will be described by referring to the accompanying drawings based on non-limiting examples. [Figure 1] A flowchart of an example of a method of saving the state of a database is shown. [Figure 2] A flowchart of an example of a method of accessing the state of a database is shown. [Figure 3] A schematic diagram of an example of changing a data page and recording the changed data page in an append-only storage is shown. [Figure 4] Examples of address specifications in a patch and shared change state mapping table, and the structure of an example of a log file are shown. [Figure 5] An example of a commit sequence of the present method is shown. [Figure 6] An example of a system is shown.

Modes for Carrying Out the Invention

[0020] Referring to the flowchart in Figure 1, a computer implementation method for saving the state of a database is proposed. This method includes step S10, which provides the database.

[0021] A "database" refers to a collection of data (i.e., information) organized for searching and retrieving (for example, a relational database based on a given structured language such as SQL). When a database is stored in memory, it allows for rapid searching and retrieval by a computer. In practice, databases are configured in combination with various data processing operations to facilitate the storage, retrieval, modification, and deletion of data. A database may consist of a file or a set of files that can be divided into records, each record consisting of one or more fields. A field is the basic unit of data storage. A field is also called a page. A database page is the basic internal structure for organizing data within a database file. A page is the basic unit of I / O operations. The size of a page varies depending on the database implementation; for example, in an SQL database server, the page size is 8kB.

[0022] Users may retrieve database data primarily through queries. They can use keywords and sorting commands to quickly search, rearrange, group, and select fields from a large number of records, and then retrieve or create reports on specific aggregations of the data according to the rules of the database management system being used.

[0023] The state of a database is the set of data stored at a particular point in time. A database is always in one specific state. Adding, deleting, or modifying the information stored in the database changes its state. For example, a database may contain information about various designs of a CAD model characterized by parameters such as length and / or shape. As an example, if one of these parameters is changed during the design process and the new set of parameters is saved to the database, its state changes.

[0024] A database can be represented as a conceptual model with several layers or levels of abstraction, each providing a conceptual model or algorithm that can be independent of a particular implementation. For example, a database may include a data structure layer and a storage layer. The storage layer (also called the physical level) is responsible for storing data in persistent memory, such as a hard drive. The storage layer may use volatile and / or persistent (i.e., non-volatile) memory to store the data. The data structure layer (also called the logical level) describes how data is stored in the database using data structures. Data structures define how data is modeled and organized for a specific purpose, such as persistent storage, and the operations and data type implementations allowed on the data are most database-specific at this level.

[0025] The method further includes step S20, which involves receiving one or more write events by the database. A write event is a request from a system communicatively coupled to the database to write information to the database, for example, a computer running a CAD application. Writing information to the database includes adding new data to the database and / or removing (i.e., deleting) information already stored in the database and / or modifying data already stored in the database. The database may allow multiple users working with the database to send write requests concurrently or substantially simultaneously. The database may queue multiple write requests and execute them in sequence. Thus, a write event is an operation that changes the state of the database after execution by the database, and execution by the database is the commit of a transaction containing a write event.

[0026] This method further includes step S30, which logs each write event received by the database. Thus, each logged write event forms a new state on the database, i.e., the commit of the write transaction containing the write event forms a new state of the database. The step of logging write events is the operation of keeping the log in non-volatile memory. It is understood that any type of non-volatile memory (also called long-term persistent storage) may be used. A log file is a file that stores a chronological record of operations performed by the database in response to requests from a system that is communicably coupled to the database. Thus, logging write events means saving the write events received by the database at time t in the log file, after the immediately preceding write event received at time t-1, in the sequence of write events received by the database. Thus, the log file contains the set of write events received by the database. The log file contains the changes (write events) applied to the saved data during the process of committing the write transaction, where the write event is within a write transaction. A write transaction is a unit of read / write events in the database that are performed while maintaining data integrity. The transaction itself consists of a set of information for updating the state of the database. The actual state of the target database can be reconstructed from the known state by applying logged changes (i.e., write events). Write events change the state of the database from the perspective of the write transaction.

[0027] This method further includes the step (S40) of holding pages modified or created by one or more write events in a buffer. As previously mentioned, database pages are the basic internal structure for organizing data in a database file, and pages are the basic unit of I / O operations. Database pages are contiguous blocks of database memory as the smallest unit of data for database management. The database architecture allocates all existing or new data structures and stores specific information about the data structures within pages. For example, a page has a physical identification in the storage hardware (where the page physically resides) and a logical identification used to ensure that the page has a fixed identifier even if the page changes over time and its physical identification differs. For example, the storage layer is the file system allocator, which functions as the data structure layer. Such mapping between logical and physical pages separates data structures from the storage layer's concurrency control model. Conceptually, it is necessary to have a consistent mapping for all pages of transactions executed on a particular identifier. As some examples, the identifier may be a timestamp, which is information encoded in a string that identifies when a particular event occurred. As another example, the identifier may be a flag indicating the state of the database (e.g., "version 1.0"). The step of holding in a buffer is to temporarily store data in a specific area of ​​physical memory called a buffer. Hereafter, this buffer will be referred to as the shared change state. These two terms may be used interchangeably and with the same meaning. The shared change state is the state of a modified page that is shared between transactions. Therefore, the shared change state is born from a transaction, can persist without a transaction, and can be combined with another shared change state to create a new shared change state, which conceptually corresponds to the combination of two transactions. Therefore, the shared change state may be linked to a transaction until the transaction is successfully committed or interrupted. If the transaction is interrupted, the shared change state may be discarded.Once the initial transaction is committed, its shared change state is no longer linked to that initial transaction and may become available to subsequent transactions. This shared change state then comes to be considered as all pages modified since the previous patch (i.e., the meaning has changed: it is no longer "pages modified by a specific transaction," but rather "pages modified since the previous patch").

[0028] The method further includes step S50, which creates a patch by flushing pages held in the buffer to database storage if a threshold is reached. The step of creating a patch means calculating the patch. The patch includes a list of new pages that have been modified or created by one or more new write events. For example, a new patch (the last calculated patch) lists the differences between the new patch and the current patch, i.e., the last patch obtained before the new patch was calculated. The act of flushing means emptying the shared change state and writing its contents to non-volatile memory. The threshold for this flush may be, for example, the size of the log and / or the size of pages held in the buffer and / or the time to rebuild from the log and / or the time elapsed since the last flush was performed. The time to rebuild from the log is the estimated time required to reapply the events recorded in the event log storage to rebuild the identification state.

[0029] This approach utilizes a hybrid choice between shadow paging and logging to improve the method for obtaining a database that combines time-travel characteristics and long-running transaction performance (authoring system) while reducing computational resources and memory generated by I / O. This hybrid strategy uses a page version mapping table combined with a memory write buffer on the one hand, and an append-only event log storage on the other. In this method, database storage is divided into event log storage containing the log and data storage containing data pages. Thus, the event log is a sequence of write events representing operations committed (or scheduled to be committed) over time, and is a purely sequential structure. Database storage represents indexed state. Therefore, with each commit of a change, data storage does not necessarily increase, but the log does.

[0030] This method can be used with various types of databases, such as graph database systems like in-memory RDF services. More specifically, the basic strategy is to implement three of the four fundamental ACID properties of a database: indivisibility, isolation, and persistence.

[0031] In particular, this method provides write transaction inseparability through an append-only storage strategy, offers independence for both read and write transactions through multi-version concurrency control while keeping I / O costs as low as possible, and provides write transaction persistence. This method does not require waiting for modified or newly created pages to be flushed to disk before accepting a transaction commit; instead, it may be based on a log of changes. This significantly reduces latency and improves overall performance. Therefore, this method provides low-cost transactions in terms of create and commit latency. The append-only strategy, where writers do not block readers and readers do not block writers, is suitable for mixed read and write workloads. These transactions can be created against any state of the database with controlled and limited latency costs, and this reduction in I / O costs can be used to "time-travel" access to the history of changes made within the database, for example, for creative authoring applications.

[0032] Furthermore, because this strategy is implemented at the database storage level rather than the data structure level, these properties are freely available to all data structures in the database. New data structures can be designed without incurring the cost of utilizing the properties described in the previous section.

[0033] This method is performed by computer. That is, the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, possibly fully automatically or semi-automatically. As an example, the initiation of at least some steps of the method may be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the need to implement the user's wishes. As an example, this level may be user-defined and / or predefined.

[0034] Figure 6 shows an example of a system, where the system is a server, for example, a server that hosts a database.

[0035] The server in this example comprises a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The server may further include a video random access memory 1100 and associated graphics processing unit (GPU) 1110 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the above may be complemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). One or more mass storage devices may be used to implement the storage layer of the database. A network adapter 1050 manages access to the network 1060. The server may also include haptic devices 1090 such as a cursor control device and a keyboard. The cursor control device is used by the server to allow the user to selectively position the cursor at any desired position on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generators for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the mouse buttons are used to generate signals. Alternatively or additionally, the server system may include a pressure-sensitive pad and / or a pressure-sensitive screen.

[0036] The computer program may include instructions that can be executed by a computer, and the instructions include means for causing the system to perform the Method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented, for example, as a device such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the Method may be performed by a programmable processor that executes a program of instructions to perform the functions of the Method by acting on input data and producing output. Thus, the processor may be programmable or coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. The application program may be implemented in a high-level procedural programming language or an object-oriented programming language, or in assembly language or machine language, as necessary. In any case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In any case, the application of the program on the system results in instructions for performing the Method.

[0037] A typical example of a computer implementing this method is running it on a system adapted for this purpose, such as a server. The system may include a memory-coupled processor, where memory stores computer programs containing instructions for performing this method. Memory may also store a database. Memory is any hardware adapted for such storage and may include several physically distinct components (e.g., one for the program and possibly one for the database).

[0038] As an example, this method is used in the process of generally manipulating modeled objects using an authoring application. A modeled object is any object defined by data stored, for example, in a database. Therefore, the expression "modeled object" refers to the data itself. Depending on the type of system, modeled objects may be defined by various types of data. The system can actually be any combination of CAD systems, CAE systems, CAM systems, PDM systems, and / or PLM systems. In these various systems, modeled objects are defined by corresponding data. Thus, CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, and CAE data can be discussed. However, since modeled objects can be defined by data corresponding to any combination of these systems, these systems are not exclusive to each other. Therefore, the system may be both a CAD system and a PLM system.

[0039] For illustrative purposes, we will discuss examples of methods used in CAD systems.

[0040] The term CAD system further refers to a system, such as CATIA, that is adapted to design modeled objects based on their graphical representation. In this case, the data defining the modeled object includes data that enables the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, and in certain cases faces or surfaces. Lines, edges, or surfaces may be represented in various ways, with non-uniform rational B-splines (NURBS) being an example. Specifically, a CAD file may contain specifications from which geometry may be generated, thereby enabling the generation of what is represented. The specifications of a modeled object may be stored in a single CAD file or multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. Also, a modeled object can typically be an assembly of thousands of parts.

[0041] In the context of CAD, modeled objects are typically 3D modeled objects, which represent products such as parts or assemblies of parts, or in some cases, product assemblies. A "3D modeled object" means an object that is modeled with data that enables a 3D representation. 3D representation allows parts to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, may be processed and rotated around any axis of its or any axis of the displayed screen. This excludes 2D icons, in particular, that are not modeled in 3D. Displaying 3D representations makes design easier (i.e., it statistically improves the speed at which designers perform tasks). Since product design is part of the manufacturing process, this speeds up the manufacturing process in industry.

[0042] The 3D modeled object may represent the geometry of a product after its virtual design has been completed, for example, by a CAD software solution or CAD system. The product may be a (e.g., mechanical) part or assembly of parts (an assembly of parts may be considered a part itself from the perspective of this method, or parts and assemblies of parts are equivalent, as this method can be applied independently to each part of an assembly), or more generally, an assembly of any rigid body (e.g., a movable mechanism). CAD software solutions enable the design of products in a wide range of industries without limitation, including aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. The 3D modeled objects designed by this method may be industrial products that are any mechanical parts, such as parts for land vehicles (e.g., automobile and light truck equipment, racing cars, motorcycles, truck and motor equipment, trucks and buses, trains, etc.), parts for aircraft vehicles (e.g., airframe equipment, aerospace equipment, propulsion equipment, defense products, aircraft equipment, space equipment, etc.), parts for marine vehicles (e.g., naval equipment, commercial ships, offshore equipment, yachts and workboats, marine equipment, etc.), general mechanical parts (e.g., industrial manufacturing machinery, heavy machinery or equipment, installation equipment, industrial equipment products, metalwork products, tire manufacturing products, etc.), electrical machinery or electronic components (e.g., home appliances, security and / or control and / or measurement products, computing and communication equipment, semiconductors, medical devices and equipment, etc.), consumer goods (e.g., furniture, home and garden products, leisure goods, fashion products, products of durable goods retailers, products of textile retailers, etc.), and packaging (e.g., food and beverages and tobacco, beauty and personal care, household goods packaging, etc.).

[0043] The CAD design process is typically collaborative, requiring multiple individuals to work independently or collaboratively. In this context, it is crucial that the CAD system's database provides all users with independence and integrity of the database state. Such design applications require creative authoring, which involves exploring various design possibilities incrementally and navigating the history of changes in a time-traveling manner. Examples of the methods of this invention are described against the backdrop of a CAD authoring application. It should be understood that these examples are not limited to CAD authoring applications but can be applied to any authoring application. More generally, these examples of the methods are not limited to a specific area of ​​authoring applications; that is, examples of the methods can be applied to any database that stores the state of the database.

[0044] Refer again to Figure 1 for further details. The database is provided in step S10. For example, this database may contain information on the CAD model design collected by one designer or multiple designers who may be working on the CAD model in parallel, through an incremental creative application. The database may include a set of data pages as the smallest unit, as mentioned above, but it should be understood that any other internal basic structure may be used to organize the data within the database file.

[0045] For example, database storage stores records in pages, and database storage is append-only. Therefore, the database is immutable, and for instance, the entire history of all transactions may be stored in event log storage. This is useful for auditing and historical queries.

[0046] Next, in step S20, one or more write events are received by the database. A write event is a user request to write information to the database, adding one or more new data or modifying one or more existing data already stored in the database. Each write event contains a set of information to update the state of the database. Write events may be sent via an application programming interface (API) or via a remote procedure call (RPC), which typically uses pipes, sockets, or shared memory. For example, if a group of designers working on the same product simultaneously modify 3D parts of an assembly of components that make up the product, each modification of the component performed by one or more designers (i.e., the received write event) generates the set of information that is applied to update the database.

[0047] Next, in step S30, each write event received by the database is logged. Logging can be performed using a persistent storage means. For example, logging is performed by maintaining one or more files (also called log files) that store a history of the changes made to the stored data during the process of at least one write event. As another example, logging may be performed using a shared log queue or a distributed file system. The log stores a history of all changes made by one or more consecutive write events, including all write events received by the database. Each logged write event creates a new state in the database, and therefore the log file stores information that allows for the recovery of the new state; in other words, the log stores the database transactions made by the received write events. It should be understood that the state in which transactions have been committed is accessible in a time-travel manner. Changes to the state that have not yet been committed are made by write operations that create a log event in memory and a new page in memory. At commit, the log event is written to disk, and the page is held in a buffer or written to disk.

[0048] For example, the step of logging each write event includes writing the history of changes made by the received write event to in-memory event log storage as an event log (in the form of an event log) for intermediate transactions, which are transactions before they are flushed to database storage (e.g., on disk). Intermediate transactions (also called un-flushed transactions) are transactions that are triggered by the received write event, are saved in the log, but are not yet associated with the persistent state of the database. In this situation, logging acts as a complete collection of all transactions made by the received write event, with one or more transactions that have not yet been flushed to the database storage of the database saved as intermediate transactions.

[0049] For example, event log storage is append-only, meaning that new logs are always appended to the event log storage rather than replacing existing data. Append-only log storage records data changes that occur by writing each change to the end of the file. In this case, the entire set of received write events can be recovered by replaying the append-only log from beginning to end. For example, one or more past write events (flushed to disk or logged) can be re-executed by replaying one or more transactions and / or one or more intermediate transactions stored in the append-only event log storage.

[0050] Next, in step S40, pages modified or created by the received write event are held in a buffer. For example, these modified or created pages are obtained as a result of user interaction with the product by the designer group, and the changes made to the product by the designer group are stored. Holding in a buffer means that pages modified or created by the received write event are stored in memory so that the system can process these pages at a later stage. Thus, modified versions of pages from write events are queued in the memory write buffer and flushed at a later stage. These modified versions of pages constitute a set of pages called the shared modified state. It should be understood that database pages are selected only for illustrative purposes, and other database fundamental units of I / O operations may be used.

[0051] These pages, modified or created by the received write events, remain in buffer memory until a threshold is reached, at which point the buffer memory is flushed in step S50. The creation of patches resulting from the flushing of buffer memory is described below.

[0052] For example, the threshold may be the size of the event log. Therefore, if the event log size exceeds or equals the threshold, the buffer memory is flushed. The event log may be a log file, as already described. The event log size may include the number of received write events stored in the event log, and / or the space occupied by the event log on the storage medium where the event log is stored (e.g., measured in megabytes (MB) or gigabytes (GB)). For example, the threshold may be set to 16MB, in which case, if the event log size exceeds 16MB, the pages held in the buffer are flushed.

[0053] For example, the threshold may be the size of the pages held in the buffer. Therefore, if the size of the pages held in the buffer exceeds or equals the threshold, the buffer memory is flushed. The size of the pages held in the buffer may include the number of pages stored in the buffer and / or the space occupied by the pages in the buffer (e.g., measured in megabytes (MB) or gigabytes (GB)). For example, the threshold may be set to 1GB, in which case, when the number of pages held and stored in the buffer equals 1GB, the pages held in the buffer are flushed.

[0054] For example, the threshold may include the time elapsed since the last flush. Thus, memory flushing is performed periodically, for example, every n seconds, n minutes, n hours, n days, n weeks, or n months.

[0055] For example, the threshold may include the time required to reconstruct from the log. Reconstruction time is an estimate of the time required to replay some or all of the logged events, and therefore reconstruction time is the time to recalculate the set of transactions that occurred or were logged. In this case, the entire set of received write events can be recovered by replaying the append-only log from beginning to end. For example, one or more past write events (whether flushed to data storage or not) are replayed by replaying one or more transactions and / or one or more intermediate transactions stored in the append-only event log storage. The reconstruction time from the last flush may be estimated based on the number of log events in the event log storage since the last flush was performed.

[0056] Step S50 includes creating patches for pages after the pages modified or created by write events stored in the buffer have been flushed to database storage. The created patches are sometimes called new patches because the system managing the database may include at least one existing patch previously obtained as a result of flushing the pages held in the buffer. Each created patch may be identifiable by being assigned an identifier (such as a timestamp or an ascending integer index), so that all created patches can be arranged in the order of creation to form a sequence of patches. In this way, the flushed patches are in chronological order of flushing by being assigned their respective identifiers. Thus, the new patch is the most recent patch created and stored at the end of the file. The new patch contains a list of one or more pages modified or created by the last received write event. The new patch lists only the page changes between the new patch and the second-to-last patch obtained as a result of the second-to-last flush of pages held in the buffer. This improves memory usage and I / O costs.

[0057] Therefore, a new patch is created each time the database receives one or more write events and reaches a threshold. Each time the creation is repeated, a list of new pages modified or created by the new write events received by the database is created. Thus, a sequence of states is obtained in which some states each have a corresponding patch. In fact, there is no guarantee that a patch represents a state, as at least the creation of a patch relies on flushing pages held in buffers to the database, triggered by reaching a threshold. Therefore, a patch can represent a state in the database, or a combination of one or more states in the database, and / or a part of one or more states in the database. States are recursively made up of patches. The set of pages Pi, which is reconstructed from the set of pages Di of patch i, is recursively reconstructed. This can be written as follows:

number

[0058] Thus, the example of this method may be repeated, for example, in the course of a CAD design process, and each time the database receives further write events and reaches a threshold, a new patch is created and flushed to database storage. Each new patch contains a list of new pages modified or created by that new write event. Each new patch may also further contain mapping tables and / or descriptors, as described later. Thus, a sequence of states is obtained in which some states each have a corresponding patch in database storage, and the corresponding states can be retrieved directly, for example, via the corresponding mapping table. Database states that do not correspond to patches in database storage are the results of transactions before they are flushed to database storage, i.e., unflushed transactions. These transactions are made persistent by being written to event log storage.

[0059] The conceptual model of a database has already been explained above. As an example, a database includes a storage layer, in which memory can be allocated, modified, and changes can be saved. Steps S30 to S50 may be implemented at the storage level. The database further includes a data structure layer, in which data structures (such as B-trees) are created by allocating memory in the storage layer and modifying that memory area to create the necessary structure. Steps S10 to S20 may be implemented at the data structure level. Figure 5 shows an example of the relationship between the data structure layer and the storage layer. Figure 5 will be explained below.

[0060] Referring to Figure 3, we illustrate a schematic example of creating a new patch. This example begins with three data pages stored in an additional-only data storage, which have already been flushed in the database. Similar to traditional shadow paging architectures, when n pages are flushed to disk, a patch of n pages is created. Each of these pages has a physical identification of the data within the storage layer where each page physically resides; for example, each page has a physical address on the storage medium, such as a disk drive. For example, the database refers to the data pages by a logical identification that is not affected by changes in physical identification; for example, each page has a logical address from the database's perspective. These three pages are "referenced" by the existing patch because they have already been flushed. For this purpose, the patch may include a mapping table. The mapping table must be predefined to establish the mapping between the logical and physical identifications described above. It should be understood that techniques other than using the mapping table may be used to access the files on disk. The patch created after the flushing of the three pages includes a mapping table. The mapping table can be considered a theoretical multi-version index. While a mapping table performs the same function as a theoretical multi-version index, it does not actually implement a full multi-version index for performance improvements. As shown in Figure 3, for timestamp T1, the mapping table maps logical addresses (1, 2, 3) to physical addresses in storage (adr#1, adr#2, adr#3), respectively. For timestamp T2, pages 1 and 3 are modified. The logical addresses remain unchanged; instead, the new mapping table points to the old physical address of page 2 at T1 and the new physical addresses of pages 2 and 3 at T2. When a transaction is created (or opened) in the database, this mapping table is constructed or instantiated in memory between the logical and physical pages. Thus, the data structure recognizes only logical pages without knowledge of the physical pages, and the data structure recognizes only logical pages without knowledge of the physical pages.If a mapping table already exists in memory, a new mapping table is created by duplicating the existing mapping table, for example, the mapping table for the previous timestamp, and applying local mappings up to the desired transaction and database state. If a mapping table does not exist, all patches are scanned by going back from descriptor to descriptor, for example, from the descriptor of the patch with timestamp T2 back to the descriptor of the patch with timestamp T1, and the latest version of all pages defined by the first descriptor selected for use is obtained. This mapping table strategy reduces the memory cost of a single multi-version index and its high lock contention. The implementation of the mapping table may choose to obtain the fastest table for associations between integers and integers / pointers. As an example, the mapping table may be created for performance efficiency based on a lock-free compare-and-swap array, as is known in the art. As another example, the mapping table may be created based on a user-space read-copy-update (RCU) synchronization mechanism.

[0061] As an example in Figure 3, a patch contains a list of one or more new pages modified or created by a new write event, and also contains descriptors. A patch representing a set of pages may be subject to several placement constraints to ensure that mapped memory is available. For example, 4k bytes per page are allocated to virtual memory on Intel (copyright) and Arm (copyright) processors. Another example: the Windows (copyright) virtual memory manager may require patches to be placed in 64k-byte blocks, and therefore patches are multiples of 64kb blocks, with descriptors written to the end of the last 64kb block. Descriptors can be added as the last element written to the end of the memory block containing the patch's pages to support indivisibility: the presence of a descriptor indicates that the pages were successfully committed; in other words, writing this descriptor last provides a simple way to verify that the entire patch is correctly registered. Otherwise, any data found after the previous descriptor is ignored, i.e., discarded. As an example, a descriptor may contain at least one of the following: - The number of physical pages modified or created by a new write event. This allows us to find the beginning of the current patch and then easily find the previous patch to pre-allocate the memory resources associated with that patch.

[0062] - Metadata to check the integrity of the descriptor, thereby ensuring that the descriptor is not corrupted.

[0063] - Timestamps of write events represented by patches. The order of patch commits (i.e., their flushes) can be obtained from the descriptors. This allows for improved database reconstruction as needed.

[0064] The presence of descriptors may provide a reliable termination condition for the page mapping algorithm. Referring again to the example in Figure 3, the patch includes a list of one or more new pages modified or created by the last received write event and a descriptor. The patch may further include a mapping table, as shown in Figure 4, which includes the mapping between the logical and physical identities.

[0065] The actual state of the target database, particularly the state that does not correspond to the patch in the database storage, can be reconstructed from the known state, i.e., the state with the corresponding patch in the database storage, in combination with the application of changes recorded in the event log files recorded in the event log storage.

[0066] Figure 2 is a flowchart illustrating an example of accessing and retrieving one of the database states stored according to the example method in Figure 1 above. In this example, according to the method, at least two database states have already been flushed from the buffer and stored on disk. Thus, a database containing at least two database states is provided in step S210. In step S220, a read event is received by the database to retrieve data about the identification state. This identification state may be queried by a user or by the computer automatically. This identification state may be a database state that already exists on disk as a result of flushing to the buffer, or a state that exists chronologically between two stored patches. In either case, the method retrieves a patch of the sequence of states from the database storage in step S221. As previously described, the patch retrieved to construct the database state includes the retrieval of preceding patches, since a patch is generally not sufficient on its own. In the former case, this retrieved patch is the corresponding patch of the database state flushed to disk as a result of a write event. In the latter case, the method retrieves a list of logged write events that occurred between the retrieved patch and the identification state. For example, the retrieved patch is the first patch that precedes the identification state in chronological order. Another example is that the retrieved patch is the first patch that follows the identification state in chronological order. Yet another example is that the retrieved patch is the patch closest to the identification state in chronological order, that is, the patch that minimizes the length of the list of logged write events that occurred between the retrieved patch and the identification state of the database. Next, in step S222, the method retrieves a list of logged write events that occurred between the retrieved patch and the identification state of the database. These log files hold a history of changes to the data pages. In step S223, the identified database is recovered by applying the list of logged write events to the retrieved patch.

[0067] Figure 5 shows an example of a commit sequence using this method. The transaction manager receives one or more write events as a series of commands 1, command 2, etc., stored in the event log. The transaction manager's role is to link the data structure to the storage layer transactions and coordinate the commits of the commands in the event log. A commit begins with the transaction manager (1) instructing the event manager to save each received event to the event log. As a result of this instruction, the events are written to the event log, which may be an append-only event log storage, and the storage layer acknowledges to the transaction manager that the events have been saved (2). Saving received write events to the event log instead of saving modified or created pages directly to the database significantly reduces I / O volume and thus reduces the potential overload on the storage layer. Steps (1) and (2) may be affected by latency, but this is not a problem because all received write events are recognized and logged by the database, and the received write events can be committed in the database at a later stage.

[0068] After a transaction is approved, the transaction manager closes the transaction. From the transaction manager's perspective, this means that the received write event is considered to have been processed by the database.

[0069] Next, the transaction manager sends a command to the storage layer (3) to save the modified pages to a buffer (4) and flush them (5) when a threshold is reached.

[0070] Each time a change is committed, the modified data pages are held in a memory buffer before being flushed to database storage, which inevitably increases the size of the event log storage, but the size of the data storage only increases with commits and subsequent flushes.

[0071] Referring to Figures 4 and 5, an example of a commit sequence using this method can be illustrated across two timestamps T1 and T2 shown in Figure 3. At timestamp T3, as shown in Figure 4, the transaction manager receives a write event command. The write event modifies data page 2, requested by thread 1. The received write event is recorded in the event log storage by the storage tier, at the order of the transaction manager.

[0072] Figure 4 shows an example of a log file. Logging is performed using known technical methods. For example, a log file contains the transaction context. The transaction context is a pseudo-object that represents its state and may include the transaction's key thread (thread 1), the transaction's start timestamp (T3), and the transaction's end timestamp. The end timestamp refers to the transaction's commit time and is calculated when the transaction is committed. Commits are serialized, meaning that transactions of changes follow one another. For example, if A starts a write transaction and B opens a write transaction, B actually starts when A's write transaction completes (e.g., is interrupted / committed), and even if B starts before A completes, B's write transaction will appear to have started after A.

[0073] Returning to Figure 5, after the correct logging of the received events (2), the transaction manager sends the modified page to a buffer (3). The transaction manager then writes the new data page 2 to the shared modified state of adr#7 in physical memory. The transaction manager also updates the mapping table corresponding to the new path with the modified page 2. This mapping table is created by first duplicating the mapping table corresponding to the patch of the last timestamp (T2) in database storage, which is read as (adr#4, adr#7, adr#5).

[0074] Returning to Figure 5, when a threshold is reached, the memory buffer is periodically flushed to disk (5). This flush threshold may be, for example, the size of the log and / or the size of the pages held in the buffer and / or the time it takes to rebuild from the log and / or the time elapsed since the last flush. As an example of this commit sequence, when the flush threshold is reached, a new patch is created by flushing the data held in the buffer (adr#7) to database storage. The new patch further includes the updated mapping tables (adr#4, adr#7, adr#5) and descriptors containing the timestamps (T3) of each patch.

[0075] Referring to Figures 4 and 5, it can be seen that this method supports indivisibility through the storage of data pages to dedicated additional storage by providing a descriptor at the end of each successfully committed patch. Furthermore, independence of read and write transactions is supported by providing different mapping tables for different versions of data pages. This design of the storage layer ensures that transactions are properly committed and their changes are visible, or not visible. In addition, all transactions are written to disk, thus providing persistence, and the corresponding state of some transactions is flushed to database storage as patches, and the corresponding state of intermediate transactions can be reconstructed using the history of changes stored in event log storage along with the saved patches. This combination of logging and periodic flushing of pages to disk supports optimized performance.

[0076] This method does not wait for modified or newly created pages to be flushed to disk before accepting the transaction commit, thus significantly reducing latency. Instead, this method recovers unsaved database states based on the change log, which is more interesting from an overall performance standpoint. This method provides low-cost transactions in terms of creation and commit latency.

[0077] Because each reader and writer uses its own mapping table and memory write buffer, the additive-only strategy, where writers do not block readers and readers do not block writers, provides optimized performance for mixed read and write workloads. These transactions can be made to any state of the database with controlled and limited latency costs, taking advantage of the reduced I / O costs. Meanwhile, by combining the nearest available data page on disk with reapplying the event log up to the desired database state, any state of the database can be accessed in virtually the same way. In the worst case, the cost of access is equal to the cost of creating the mapping table in the memory-mapped data file plus the cost of replaying the log that generated the number of pages necessary to reach the threshold for flushing pages on disk. Thus, this method provides a time-traveling access capability to the history of changes made within the database for creative authoring applications.

[0078] Furthermore, because this strategy is implemented at the database storage level rather than the data structure level, these properties are freely available to all data structures in the database. New data structures can be designed without incurring any costs to utilize the properties shown by this method.

[0079] Preferred embodiments of the present invention have been described. It is understood that various modifications can be made without departing from the spirit and scope of the invention. Therefore, other implementations are also within the scope of the following claims.

Claims

1. A computer implementation method for saving the state of a database, Step (S10) provides the initial state of the database in an identifiable manner, The steps include: receiving one or more write events from the database (S20), A step (S30) of logging each write event, wherein each logged write event forms a new state in the database (S30), Step (S40): The page modified or created by the write event is stored in a buffer, and a mapping table is constructed in the buffer that associates the physical identification information of each stored page with the logical identification information of the page in the data structure layer of the database. When at least one of the following reaches a threshold: the size of the event log, the size of the pages held in the buffer, the time taken to rebuild from the log, and the time elapsed since the last flush, the pages held in the buffer and the constructed mapping table are flushed to database storage. (a) A list of new pages that have been modified or created by the write event, (b) The mapping table flushed from the buffer, and (c) Time-series information showing the patch's timeline Step (S50) of creating a patch that includes and has a data structure that lists only the page changes between it and the preceding patch, Starting from the provided database, the identification state of the database is restored by applying the list of write events logged after the time-series information corresponding to the created patch to the created patch, using the mapping table included in the created patch. (S60) Computer implementation methods, including those mentioned above.

2. The step further includes repeating the creation step each time the threshold is reached after the database receives a new write event, Each iteration of the creation step involves recursively combining each created patch with previous patches to obtain a sequence of database states, The computer implementation method according to claim 1.

3. Each logged write event has a local mapping table obtained by replicating the mapping table constructed in the buffer, updating the mapping up to the target logged write event, and applying the local mapping. The computer implementation method according to claim 1 or 2.

4. The aforementioned mapping table is created by a lock-free compare-and-swap operation. The computer implementation method according to claim 3.

5. The created patch further includes a descriptor, which is the data element written at the end of the patch. The descriptor is, The number of physical pages modified or created by a new write event. Metadata to verify descriptor integrity, Timestamp of a new write event, It includes at least one of the above, detects that the descriptor was written at the end of the patch, and uses the metadata to check whether the descriptor itself is corrupted, thereby verifying the integrity of whether the patch was written completely to the end. The computer implementation method according to any one of claims 2 to 4.

6. When flushing to the database storage, the descriptor is written to the end of the patch, so that the presence of the descriptor at the end of the patch indicates that the pages of the patch have been successfully committed, and if the descriptor is not present, the successful commit has failed and the corresponding logged write event is discarded. The computer implementation method according to claim 5.

7. The database storage is for appending only. The computer implementation method according to any one of claims 1 to 6.

8. The step of logging each of the aforementioned write events includes writing a history of changes as an event log to log storage on disk for intermediate transactions, which are transactions before flushing to the database storage. The computer implementation method according to any one of claims 1 to 7.

9. The log storage is add-only. The computer implementation method according to claim 8.

10. A computer implementation method for accessing the state of a database, Step (S210) provides a database in which the patch created according to the method of any one of claims 1 to 9 is stored in the database storage and the logged write events are stored in the event log storage. The step of receiving a read event from the database for the identification state, which is the state of the database designated as the target of recovery, Step (S221): Obtain from the database storage the patch that is closest to the identification state among the patches adjacent to the identification state in time series. Step (S222): Obtain a list of logged write events that occurred between the acquired patch and the identification state from the event log storage. Step (S223): Starting from the provided database, the identification state of the database is restored by applying the list of logged write events to the acquired patch. Computer implementation methods including

11. The acquired patch is a patch that minimizes the length or size of the list of logged write events that occurred between the acquired patch and the identification state of the database. The computer implementation method according to claim 10.

12. A computer program comprising instructions for performing the computer implementation method described in any one of claims 1 to 9.

13. A computer program comprising instructions for performing the computer implementation method according to claim 10 or 11.

14. A computer system comprising a database, a computer-readable storage medium recording the computer program described in claim 12 or 13, and a computer that reads the computer program from the computer-readable storage medium and executes it on the database.

15. The database further includes a transaction manager that links the data structure to transactions in the storage layer and coordinates commits. The transaction manager supplies the event log to the storage layer, The persistence of the event logs on log storage is authorized by the storage layer, After approval, the transaction manager closes the transaction in the storage layer and sends the modified pages to the memory buffer. The computer system according to claim 14.

Images