Parallel loading of mapping containers for database system startup and restart operations

Abstract

Aspects of the current subject matter relate to methods in which parallel loading operations of file ID mapping containers are completed at startup and / or restart of a database system. Parallel loading operations of file ID mappings and / or large binary object file ID mappings into multiple data buffers associated with each of a plurality of scan engines are completed between the plurality of scan engines. Each scan engine operates on a specific path of a page chain comprising a mapped page structure, such that the page chain is split between the scan engines to process the mapping. The contents of the data buffers are pushed to the mapping engine via a queue. The mapping engine loads the file ID mapping and the LOB file ID mapping into the mapping for intra-system access.

Description

Technical Field

[0001] The present disclosure relates generally to improving startup and restart times of database systems, and more particularly, to parallel loading operations of mapping containers for improving startup and restart times of database systems. Background Art

[0002] Database management systems have become an integral part of many computer systems. Some systems process hundreds or even thousands of transactions per second, and some perform very complex multi-dimensional analysis of data. The length of startup and restart time for such systems is a critical factor affecting customers. Summary of the invention

[0003] Methods, systems, and articles of manufacture, including computer program products, are provided for parallel loading operations of a file identifier (ID) mapping container for startup and / or restart of a database system, such as a database cloud system.

[0004] According to aspects, a computer-implemented method includes: scanning, by a first scanning engine, first data from a remote memory into a plurality of buffers, wherein each of the plurality of buffers receives a dedicated portion of the first data; in response to determining that a first buffer of the plurality of buffers is full, loading a first dedicated portion of the first data from the first buffer into a first memory queue, wherein a database system includes a plurality of buffers and a first memory queue; and in response to determining that the first memory queue is full, transmitting, by a first mapping engine, the contents of the first memory queue to a first mapping stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of the second data from the second scanning engine, wherein the first data and the second data include mapping data.

[0005] According to interrelated aspects, the system includes at least one data processor and at least one memory storing instructions that, when executed by the at least one data processor, result in operations including: scanning, by a first scanning engine, first data from a remote memory into a plurality of buffers, wherein each of the plurality of buffers receives a dedicated portion of the first data; in response to determining that the first buffer is full, loading a first dedicated portion of the first data from a first buffer of the plurality of buffers into a first memory queue, wherein the database system includes the plurality of buffers and the first memory queue; and in response to determining that the first memory queue is full, transferring, by a first mapping engine, the contents of the first memory queue to a first mapping stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of the second data from the second scanning engine, wherein the first data and the second data include mapping data.

[0006] According to an interrelated aspect, a non-transitory computer-readable storage medium includes program code that, when executed by at least one data processor, causes operations including: scanning, by a first scanning engine, first data from a remote memory into a plurality of buffers, wherein each of the plurality of buffers receives a dedicated portion of the first data; in response to determining that the first buffer is full, loading a first dedicated portion of the first data from a first buffer of the plurality of buffers into a first memory queue, wherein a database system includes the plurality of buffers and the first memory queue; and in response to determining that the first memory queue is full, transferring, by a first mapping engine, the contents of the first memory queue to a first mapping stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of the second data from the second scanning engine, wherein the first data and the second data include mapping data.

[0007] In some variations, one or more features disclosed herein including the following features may be optionally included in any feasible combination. The remote memory may be external to the database system, and the first scan engine and the second scan engine may be connected to the remote memory. The mapping data may include file identifier mapping data and / or large binary object file identifier mapping data. The first memory queue may include a single-in, full-out queue. The second dedicated portion of the first data may be loaded from a second buffer of the plurality of buffers into the second memory queue, and in response to a determination that the second buffer is full; and in response to a determination that the second memory queue is full, the second mapping engine may transfer the contents of the second memory queue to a second mapping stored in the memory of the database system. The contents of the second memory queue may include a second dedicated portion of the first data from the first scan engine and at least a second dedicated portion of the second data from the second scan engine. The loading of the first dedicated portion of the first data and the loading of the second dedicated portion of the first data may be completed in independent parallel threads. The second dedicated portion of the first data may be loaded from a second of the plurality of buffers into the second memory queue, and in response to a determination that the second of the plurality of buffers is full; and in response to a determination that the second memory queue is full, the contents of the second memory queue may be transferred by the first mapping engine to the first mapping stored in the memory. The contents of the second memory queue may include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine. The database system may receive a request from a client device related to access of mapping data; and the request may be processed by the database system by accessing at least the mapping data in the first mapping stored in the memory. The dedicated portion of the first data may be based on one or more attributes of the first data.

[0008] It should be understood that the foregoing overview and the following detailed description are exemplary and explanatory only and are not restrictive. In addition to the features and / or variations set forth herein, further features and / or variations may also be provided. For example, the embodiments described herein may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several additional features disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed embodiments.

[0010] Figure 1 is a system diagram illustrating a data storage system in which embodiments of the current subject matter may be employed;

[0011] Figure 2is a block diagram illustrating features of a parallel load engine consistent with embodiments of the present subject matter;

[0012] Figure 3 is a block diagram illustrating aspects of a parallel load operation consistent with embodiments of the present subject matter;

[0013] Figure 4 depicts a flow chart illustrating a process for a parallel load operation consistent with an implementation of the present subject matter;

[0014] Figure 5 is a schematic diagram illustrating an exemplary system including a data storage application according to some implementations of the current subject matter;

[0015] Figure 6 It is shown Figure 5 a schematic diagram of the details of the system; and

[0016] Figure 7 A block diagram illustrating a computing system consistent with implementations of the current subject matter is depicted.

[0017] Like labels are used to refer to the same or similar items in the drawings. DETAILED DESCRIPTION

[0018] Various aspects of the current subject matter relate to loading a mapping container, such as a file identifier (ID) mapping container, at system startup or restart of a database system. In some database systems, such as cloud database systems, a large portion of the startup and restart time can be attributed to loading the mapping container from persistent storage (e.g., remote persistent storage, such as disk-based storage) into system internal memory. The mapping container may include directory information required for the operation of the database system, which may be in the form of a hierarchical structure of directories and files. The mapping container may include a page chain that needs to be read at startup and / or restart and populates different transient mappings to provide a transient index. For example, the mapping container may include a file ID mapping and a large binary object file ID mapping required to access files and LOBs. Database systems with a large number of virtual file LOBs or medium-sized LOBs may affect the file ID mapping and / or LOB file ID mapping and contribute to the startup and restart time due to the amount of loading mapping that needs to be completed at startup and restart.

[0019] Existing solutions address the loading of mapping containers by scanning the file ID map and the LOB file ID map in a single thread during startup. The scan operation creates a transient picture of the database system's operations (e.g., including online operations that access files and LOBs). However, traditional mapping scans and builds can result in high load times, especially in database systems with a large number of virtual file LOBs or medium-sized LOBs. In some cases, database systems may contain billions of LOBs, resulting in load times of up to hours. The high load times are caused by the single-threaded nature of loading data from mapping containers into the mapping; one thread is used to scan the page structure, and for each entry in the page chain, information is added to the mapping.

[0020] Embodiments of the present subject matter are directed to methods in which parallel loading operations of file ID mapping containers are completed at startup and / or restart of a database system such as a database cloud system. Specifically, aspects of the present subject matter provide operations for loading file ID mappings and LOB file ID mappings between multiple scan engines in parallel into multiple data buffers associated with each of the multiple scan engines. Each scan engine operates on a specific path of a page chain, causing the page chain to be split between the scan engines to process the mapping. The contents of the data buffer are pushed to the mapping engine via a queue. The mapping engine loads the file ID mapping and the LOB file ID mapping into the mapping for in-system access. More details are provided herein.

[0021] Although the description provided herein may refer to file ID mapping and LOB file ID mapping, the implementation is not limited to this particular type of mapping data. For example, the implementation of the current theme may be applicable to various mapping containers that need to be loaded from disk to system internal memory. In addition, although the description provided herein may refer to both file ID mapping and LOB file ID mapping, the implementation is not limited to these two types of mapping data.

[0022] Figure 1 A system diagram illustrating an exemplary data storage system 100 according to some implementations of the current subject matter is shown. System 100 may include a database management system (DBMS) 106, remote storage 108 (e.g., persistent storage), and a user / client 102. In some implementations, remote storage 108 may be configured to store file ID mappings and large binary object (LOB) file ID mappings in, for example, a file ID mapping container 116.

[0023] The remote storage 108 may be communicatively coupled to a database management system 106, which may include a parallel load engine 110 and an in-memory storage 112. The in-memory storage 112 may include a map 114, which includes a file ID map and a LOB file ID map, which are loaded from a file ID map container 116 of the remote storage 108 via the parallel load engine 110. The database management system 106 may be configured to respond to requests from one or more users / clients 102 of the data storage system 100. The users / clients 102 may communicate with the database management system 106 via a network 104, which may be any wired and / or wireless network, including, for example, a public land mobile network (PLMN), a wide area network (WAN), a local area network (LAN), a virtual local area network (VLAN), the Internet, etc. The users / clients 102 may send a request to the database management system 106 to access one or more files and / or LOBs whose locations are contained in the map 114 of the in-memory storage 112. Consistent with implementations of the current subject matter, parallel load engine 110 implements a process for loading file ID maps and LOB file ID maps into map 114. Loading of file ID maps and LOB file ID maps from remote storage 108 to in-memory storage 112 may occur at startup time and / or restart time of database management system 106.

[0024] Figure 2 2 is a block diagram 200 illustrating features of a parallel load engine 110 consistent with an embodiment of the present subject matter. The parallel load engine 110 is connected to a remote memory 108 to access file ID maps and LOB file ID maps in a file ID map container 116. Consistent with an embodiment of the present subject matter, the file ID map and the LOB file ID map may be in the form of a page structure including a page chain. The parallel load engine 110 transmits the file ID map and the LOB file ID map to a map 114 in an in-memory storage 112 of a database management system 106 during a parallel load operation at system startup and / or restart. The transmission of the file ID map and the LOB file ID map allows processing of requests (e.g., from a user / client 102) related to access to files associated with the file ID map and / or LOBs associated with the LOB file ID map. In some embodiments, each map includes files and / or LOBs of a particular type or types. For example, a map may include LOBs that belong to or have particular attributes.

[0025] like Figure 2 As shown, the parallel loading engine 110 includes a scanning engine 202, a buffer 204, a queue 206, a mapping engine 208, and a map 210. Scanning engines A (202a), B (202b), and N (202n) are shown, each having three corresponding buffers B1 (204a-1), B2 (204a-2), B3 (204a-3); B1 (204b-1), B2 (204b-2), B3 (204b-3); and B1 (204n-1), B2 (204n-2), B3 (204n-3). Queues A (206a), B (206b), and N (206n) are also shown, which have respective mapping engines A (208a), B (208b), and N (208n) and respective maps A (210a), B (210b), and N (210n). Figure 2 The number of scan engines 202, buffers 204, queues 206, mapping engines 208, and maps 210 shown in FIG. 2 is purely exemplary and is used for illustration purposes only.

[0026] According to aspects of the current subject matter, each scan engine 202 reads (e.g., scans) a corresponding portion (e.g., a portion of a page structure) of a file ID map and / or a LOB file ID map contained in a file ID map container 116 from remote memory 108. In some embodiments, each scan engine 202 can read an equal portion of a file ID map and / or a LOB file ID map. Various parameters related to, for example, the size and / or type of the file ID map and / or the LOB file ID map and / or the capabilities of the scan engine 202, buffer 204, queue 206, and / or mapping engine 208 can be used to determine the number of scan engines 202 included in the parallel load engine 110. This in turn can be used to determine the distribution of the file ID map and / or the LOB file ID map between the scan engines 202. Consistent with embodiments of the current subject matter, the number of scan engines 202 can be determined by the number of pages in the page chain (which is a previously known value). The distribution of data within the map can be used to determine the striping of the map 210. According to aspects of the current subject matter, map 210 may also be striped, resulting in, for example, one queue (eg, 206a) and one map engine (208a) for each stripe of a map (eg, map A 210a).

[0027] Each scanning engine 202 is associated with a plurality of buffers 204. The scanning engine 202 transmits a portion of the read (e.g., scanned) file ID map and / or LOB file ID map to each buffer 204. That is, the file ID map and LOB file ID map read by the scanning engine A 202a are distributed in buffers B1 (204a-1), B2 (204a-2), and B3 (204a-3).

[0028] Consistent with implementations of the present subject matter, queues A (206a), B (206b), and N (206n) each receive data (e.g., file ID mappings and / or LOB file ID mappings) from one buffer 204 associated with each of the scan engines 202. For example, data from queue A 206a includes buffer data from buffers B1 204a-1, B1 204b-1, and B1 204n-1. In accordance with aspects of the present subject matter, queue 206 can be a single-in all-out (SIAO) queue. SIAO queues allow data to be transmitted in separate and / or independent segments or fragments, while all data contained in the queue is transmitted out in one stream. In accordance with aspects of the present subject matter, the amount inserted into queue 206 can be adjusted by transmitting more or less data from buffer 204 to the queue.

[0029] Consistent with implementations of the current subject matter, once a particular buffer 204 has been filled, the data from that buffer 204 is moved to its corresponding queue 206. Data continues to be scanned by the scan engines 202 into the buffers 204 of all buffers 204 and then pushed into the queues 206. This movement of data results in a parallel load transfer of data between the queues 206. Specifically, each of the scan engines 202 receives a portion of the data that is then separated in the buffers 204 and then distributed in the queues 206, wherein each of the scan engines 202 operates as their own independent thread.

[0030] Each of the queues 206 is associated with a corresponding mapping engine 208. The mapping engine 208 consumes the data contained in the queue 206 and fills the map 210 with the consumed data. The mapping engine 208 is associated with a corresponding map 210. Each map 210 may include files and / or LOBs of a specific type or types. For example, a map may include LOBs that belong to or have a specific attribute or attribute set. In some embodiments, the number of maps 210 of the database management system 106 is set based on, for example, the number and / or type of files and / or LOBs. The scanning engine 202 can be operated to move specific segments of the page chain (e.g., file ID mapping and LOB file ID mapping) based on one or more attributes associated with the mapping. That is, the scanning engine 202 operates to transfer data so that the file ID mapping and LOB file ID mapping having a specific one or more attributes are moved to the corresponding map 210. According to aspects of the current subject matter, the number of maps 210 is fixed and based on, for example, the type of information required for various operations on files and / or LOBs.

[0031] Figure 3 304a is a block diagram illustrating aspects of a parallel load operation 300 consistent with implementations of the current subject matter. According to aspects of the current subject matter, a page structure 302 including a file ID map and / or a LOB file ID map can be a page chain and can be allocated as a first portion 304a, a second portion 304b, a third portion 304c, etc. The page structure 302 can be stored in a file ID map container 116 of the remote memory 108.

[0032] Each scan engine 202 is associated with a particular portion of the page structure 302 and operates to scan the associated portion 304 and distribute it in the buffer 204 associated with the scan engine 202. For example, Figure 3 As shown, scanning engine A 202a scans the first portion 304a and distributes it in buffers B1 (204a-1), B2 (204a-2), and B3 (204a-3). The distribution of the first portion 304a in buffers 204 can be based on one or more attributes of the file ID mapping and the LOB file ID mapping. For example, the file ID mapping and the LOB file ID mapping can be split or distributed in buffers 204 so that mappings with similar attributes go to the same mapping 210.

[0033] When a particular buffer 204 reaches a full state, containing a dedicated share or portion of data from the scan engine 202 that forms part of a page structure (e.g., a page chain), the contents of the buffer 204 are moved to its corresponding queue 206. For example, the contents of the buffer 204 may be moved to a particular queue 206 that is associated with a mapping 210, which in turn is associated with one or more attributes that are the same as the attributes of the file ID mapping and the LOB file ID mapping in the buffer 204. Consistent with embodiments of the present subject matter, each buffer 204 is dedicated to a particular one of the queues 206 and a corresponding one of the mappings 210.

[0034] Aspects of the current subject matter provide for parallel data transfers between buffers 204 and queues 206. When a buffer 204 is full, its contents are transferred to its corresponding queue 206, allowing the scan engine 202 to add more data to the buffer 204. This process is repeated in parallel among the scan engine 202, buffers 204, and queues 206.

[0035] Continue to refer Figure 3 , the contents of buffers B1 204a-1, 204b-1, and 204c-1 are transferred to queue A 206a, while the contents of buffers B2 204a-2, 204b-2, and 204c-2 are transferred to queue B 206b, and the contents of buffers B3 204a-3, 204b-3, and 204c-3 are transferred to queue C 206c. While buffers 204 are queued at queue 206, mapping engine 208 (in parallel, a separate thread) consumes the contents of queue 206. This frees up queue 206, allowing data from buffers 204 to be transferred to queue 206 until page structures 302 are properly allocated in mapping 210.

[0036] According to aspects of the current subject matter, no synchronization is required on the buffers as each scan engine (e.g., scan engine 202a) only fills its local buffers (in this example, buffers B1 204a-1, B2 204a-2, and B3 204a-3). Each buffer (in this example, buffer B2 204a-2) is assigned to one of the mappings (in this example, mapping B 210b), which is filled by the mapping engine (in this example, mapping engine B 208b) through its associated processing queue (in this example, queue B 206b). The only point of contact between the scan engine and the mapping engine is the SIAO queue (in this example, queue B 206b) implemented by, for example, a cloud application service (CAS) operation. If the buffers are full, the scan engine can add a single buffer to the queue, and the mapping engine will retrieve the complete list of buffers after it has processed the previous buffer.

[0037] In some implementations, multiple queues 206 and multiple mapping engines 208 may be associated with a given mapping 210. For example, a mapping 210 may be loaded by two or more pairs of queues 206 and mapping engines 208, rather than being associated with a single mapping engine 208 and queue 206. The loading of a mapping 210 may be accomplished as a parallel, independent thread for each of the mapping engines 208 with which it is associated.

[0038] Figure 4 A flowchart 400 illustrating a process for a parallel load operation consistent with embodiments of the current subject matter is depicted.

[0039] At 410, the first scan engine may scan the first data from the remote memory into a plurality of buffers. For example, the scan engine A 202a may scan the first portion of the page chain of the data from the remote memory 108 including the file ID mapping container 116 into the buffers B1 204a-1, B2 204a-2, and B3 204a-3. The first portion of the page chain of the data may be the first portion 304a of the page structure 302, and may include the file ID mapping and the LOB file ID mapping to be inserted into the mapping 210 to provide for database operations that require mapping data. According to various aspects of the current subject matter, the buffer 204 receives a dedicated portion of the first data (e.g., the first portion of the page chain). For example, the dedicated portion may be based on one or more attributes of the first portion of the data. In some cases, the distribution of the first portion 304a in the buffer 204 may be based on one or more attributes of the file ID mapping and the LOB file ID mapping. For example, the file ID mapping and the LOB file ID mapping may be segmented or distributed in the buffer 204 so that mappings with similar attributes are designated and routed to the same mapping 210.

[0040] At 420, a first dedicated portion of the first data may be loaded into the first memory queue. For example, the first dedicated portion may be from the first buffer B1 204a-1 and may be loaded into the memory queue A 206a when, for example, the first buffer B1 204a-1 is full. Consistent with implementations of the current subject matter, data transfers between buffers 204 and to the page structures 302 of the queues 206 may occur in parallel but independent threads. For example, the contents of buffer B1 204a-1 may be loaded into the queue A 206a at a different time than the contents of buffer B1 204b-1, and both may be loaded into the queue A 206a at a different time than the contents of buffer B1 204c-1.

[0041] At 430, the first mapping engine may transfer the contents of the first memory queue into the first mapping. For example, mapping engine A 208a may consume the contents of memory queue A 206a into mapping 210a in response to a determination that memory queue A 206a is full. Consistent with implementations of the current subject matter, the contents of memory queue A 206a may include a first dedicated portion of first data from the first scan engine and at least a first dedicated portion of second data from the second scan engine (e.g., a second portion of the page chain). For example, the contents of buffers B1 204a-1, 204b-1, and 204c-1 are transferred to queue A 206a. When queue A 206a is full, mapping engine A 208a consumes the contents of queue A 206a into mapping 210a. According to aspects of the current subject matter, a given queue 206 may not have an equal amount of data from each of its associated buffers 204 at a given time. Rather, when each buffer 204 is full, the contents of the associated buffer 204 are added to the queue 206.

[0042] Similarly, the contents of buffers B2 204a-2, 204b-2, and 204c-2 are transferred to queue B 206b. When queue B 206b is full, mapping engine B 208b consumes the contents of queue B 206b into map 210b.

[0043] Consistent with implementations of the current subject matter, loading of the first dedicated portion of the first data (eg, the first portion of the page chain) and loading of the second dedicated portion of the first data (eg, the first portion of the page chain) are performed in separate parallel threads.

[0044] Aspects of the current subject matter also provide for receiving a request related to access of mapping data by database management system 106 from user / client 102. For example, user / client 102 may initiate an operation requesting a file and / or LOB to be mapped according to a file ID mapping and / or a LOB file ID mapping in mapping 114. In response to the request, database management system 106 may access mapping data in a mapping, such as first mapping 210a.

[0045] Thus, aspects of the current subject matter advantageously provide a parallel load operation method for mapping file IDs and LOB file IDs between multiple scan engines into multiple data buffers associated with each of the multiple scan engines. Each scan engine operates on a specific path of a page chain, causing the page chain to be split between the scan engines to process the mapping. The contents of the data buffers are pushed to the mapping engine via a queue. The mapping engine loads the file ID mapping and LOB file ID mapping into the mapping for access within the system. The parallel but independent data transfer operations between the scan engines, buffers, memory queues, and mappings provide reduced startup time and / or restart time of the database management system.

[0046] In some embodiments, the current subject matter may be implemented in various in-memory database systems such as those provided by the Waldorf, Germany A high performance analytical appliance ("High Performance Analytic Appliance, HANA") system developed. For example, various systems, such as an enterprise resource planning ("enterprise resource planning, ERP") system, a supply chain management ("supply chain management, SCM") system, a supplier relationship management ("supplier relationship management, SRM") system, a customer relationship management ("customer relationship management, CRM") system, and / or other systems, may interact with the in-memory system for the purpose of accessing data. Other systems and / or combinations of systems may be used in embodiments of the current subject matter. The following is a discussion of an exemplary in-memory system.

[0047] Figure 5 An exemplary system 500 is shown in accordance with some implementations of the current subject matter, in which a computing system 502, which may include one or more programmable processors, etc., configured and linked via one or more networks, may execute one or more modules, software components, etc. of a data storage application 504. The data storage application 504 may include a database, an enterprise resource program, a distributed storage system (e.g., available from Sunnyvale, California) or a computer system 502. The company acquired one or more of NetApp Filer).

[0048] One or more modules, software components, etc. may be accessed by local users of the computing system 502 and by remote users accessing the computing system 502 from one or more clients 506 via a network connection 510. One or more user interface screens generated by the one or more first modules may be displayed to the user via a local display or via a display associated with one of the client machines 506. Data units of the data storage application 504 may be temporarily stored in a persistence layer 512 (e.g., a page buffer or other type of temporary persistence layer), which may write the data to one or more stores 514 in the form of storage pages, for example, via an input / output component (I / O) 516. The one or more stores 514 may include one or more physical storage media or devices (e.g., hard drives, persistent flash memory, random access memory, optical media, magnetic media, etc.) configured to write data for longer term storage. It should be noted that the storage device 514 and the input / output component 516 may be included in the computing system 502, although they are not necessarily stored in the storage device 514. Figure 5 502 is shown as being external to computing system 502.

[0049] The data stored in long-term storage 514 can be organized in pages, each of which is allocated a specified amount of storage space. In some embodiments, the amount of storage space allocated to each page can be constant and fixed. However, other embodiments in which the amount of storage space allocated to each page can vary are also within the scope of the current subject matter.

[0050] Figure 6 An exemplary software architecture 600 is shown in accordance with some implementations of the current subject matter. A data storage application 504, which may be implemented in one or more hardware and software, may include one or more of a database application, a network-attached storage system, and the like. In accordance with at least some implementations of the current subject matter, such a data storage application 504 may include a persistence layer 512 or other type of memory buffer, or otherwise interface with a persistence layer 512 or other type of memory buffer, such as via a persistence interface 602. A page buffer 604 within the persistence layer 512 may store one or more logical pages 606, and may optionally include shadow pages, active pages, and the like. The logical pages 606 retained in the persistence layer 512 may be written to a memory (e.g., longer-term storage, etc.) 514 via an input / output component 516, which may be a software module, a subsystem implemented in one or more software and hardware, and the like. The storage 514 may include one or more data volumes 610, wherein the stored pages 612 are allocated in physical memory blocks.

[0051] In some implementations, the data storage application 504 may include or otherwise communicate with a page manager 614 and / or a savepoint manager 616. The page manager 614 may communicate with a page management module 620 at the persistence layer 512, which may include a free block manager 622 that monitors page state information 624, such as the state of physical pages within the storage 514 and logical pages in the persistence layer 512 (and optionally in the page buffer 604). The savepoint manager 616 may communicate with a savepoint coordinator 626 at the persistence layer 512 to handle savepoints, which are used to create a consistent and persistent state of the database for restart after a possible crash.

[0052] In some implementations of the data storage application 504, the page management module of the persistence layer 512 can implement shadow paging. A free block manager 622 within the page management module 620 can maintain the state of the physical page. The page buffer 604 can include a fixed page state buffer that operates as discussed herein. A converter component 640, which can be part of the page management module 620 or in communication with the page management module 620, can be responsible for writing the mapping between the logical and physical pages of the storage 514. The converter 640 can maintain the current mapping of the logical page to the corresponding physical page in the converter table 642. The converter 640 can maintain the current mapping of the logical page 606 to the corresponding physical page in one or more converter tables 642. When the logical page 606 is read from the storage 514, the converter 640 can be used to look up the storage page to be loaded from the one or more converter tables 642. When the logical page is written to the storage 514 for the first time after the save point, a new free physical page is assigned to the logical page. The free block manager 622 marks the new physical page as "used", and the new mapping is stored in the one or more converter tables 642.

[0053] The persistence layer 512 can ensure that the changes made in the data storage application 504 are persistent and that the data storage application 504 can be restored to the most recently submitted state after restart. Writing data to the storage 514 does not need to be synchronized with the end of the write transaction. Therefore, when the write transaction is completed, uncommitted changes may be written to disk, while committed changes may not yet be written to disk. After a system crash, changes made by unfinished transactions may be rolled back. Changes that have occurred in transactions that have been submitted should not be lost in the process. A recorder component 644 can also be included to store changes made to the data of the data storage application in a linear log. The recorder component 644 can be used during recovery to replay operations since the last save point to ensure that all operations are applied to the data, and before rolling back the unfinished transactions at the end of the recovery process, the transaction with a recorded "commit" record is committed.

[0054] For some data storage applications, writing data to disk is not necessarily synchronized with the end of the write transaction. It may happen that uncommitted changes are written to disk, and at the same time, when the write transaction completes, the committed changes have not yet been written to disk. After a system crash, the changes made by the unfinished transactions must be rolled back, and the changes made by the committed transactions must not be lost.

[0055] To ensure that committed changes are not lost, logging component 644 may write redo log information whenever a change is made. This information may be written to disk at the latest at the end of a transaction. Log entries may be stored in a separate log volume while normal data is written to the data volume. Using the redo log, committed changes may be recovered even if the corresponding data page has not been written to disk. To undo uncommitted changes, persistence layer 512 may use a combination of undo log entries (from one or more logs) and shadow paging.

[0056] Persistent interface 602 can handle read and write requests for storage (store) (e.g., in-memory storage, etc.). Persistent interface 602 can also provide a write method for writing data including both recording and not recording. If a recorded write operation is used, persistent interface 602 calls recorder 644. In addition, recorder 644 provides an interface that allows storage (e.g., in-memory storage, etc.) to directly add log entries to the log queue. The recorder interface also provides a method for requesting that the log entries in the in-memory log queue be flushed to disk.

[0057] The log entry contains a log sequence number, a log entry type, and a transaction identifier. Depending on the type of operation, the logger 644 records additional information. For example, for an entry of type "update", this would be the identification of the affected record and the after image of the modified data.

[0058] When the data application 504 is restarted, the log entries need to be processed. To speed up this process, the redo logs are not always processed from the very beginning. Instead, as described above, savepoints may be performed periodically to write to disk all changes made since the last savepoint (e.g., in memory, etc.). When the system is started, only the logs created after the last savepoint need to be processed. After the next backup operation, old log entries before the savepoint position may be deleted.

[0059] When logger 644 is called to write a log entry, it does not write to disk immediately. Instead, it may place the log entry into a log queue in memory. The entries in the log queue may be written to disk at the latest when the corresponding transaction completes (commits or aborts). To ensure that committed changes are not lost, a commit operation will not complete successfully until the corresponding log entry is flushed to disk. Writing log queue entries to disk may also be triggered by other events, such as when a log queue page is full or a savepoint is executed.

[0060] For the current topic, logger 644 can write database logs (or simply referred to as "logs" in this article) sequentially into a memory buffer in a natural order (e.g., sequential, etc.). If several physical hard disks / storage devices are used to store log data, several log partitions can be defined. Thereafter, logger 644 (which is used to generate and organize log data as described above) can load balance writes to the log buffer across all available log partitions. In some cases, load balancing is performed according to a round-robin allocation scheme in which various write operations are directed to the log buffer in a sequential and continuous manner. With this arrangement, the log buffer for a single log segment of a particular partition of a multi-partition log is not continuous. However, during recovery, the log buffer can be reordered to the correct order from the log segments of all partitions.

[0061] As described above, the data storage application 504 can use shadow paging so that the savepoint manager 616 can write transactionally consistent savepoints. With this arrangement, the data backup includes a copy of all data pages included in a particular savepoint, which is completed as the first step of the data backup process. The current subject matter can also be applied to other types of data page storage.

[0062] Figure 7 A block diagram illustrating a computing system 700 consistent with implementations of the current subject matter is depicted. In some implementations, the current subject matter can be configured to be implemented in system 700.

[0063] like Figure 7As shown, computing system 700 may include processor 710, memory 720, storage device 730 and input / output device 740. Processor 710, memory 720, storage device 730 and input / output device 740 may be interconnected via system bus 750. Processor 710 is capable of processing instructions executed within computing system 700. Such executed instructions may implement, for example, one or more components of system 100. In some embodiments of the current subject matter, processor 710 may be a single-threaded processor. Alternatively, processor 710 may be a multi-threaded processor. Processor 710 is capable of processing instructions stored on memory 720 and / or storage device 730 to display graphical information of a user interface provided via input / output device 740.

[0064] Memory 720 is a computer-readable medium, such as a volatile or non-volatile medium, that stores information within computing system 700. For example, memory 720 may store a data structure representing a configuration object database. Storage device 730 can provide persistent storage for computing system 700. Storage device 730 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, or other suitable persistent storage device. Input / output device 740 provides input / output operations for computing system 700. In some embodiments of the current subject matter, input / output device 740 includes a keyboard and / or a pointing device. In various embodiments, input / output device 740 includes a display unit for displaying a graphical user interface.

[0065] According to some implementations of the current subject matter, the input / output device 740 can provide input / output operations for network devices. For example, the input / output device 740 can include an Ethernet port or other network port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0066] In some embodiments of the current subject matter, computing system 700 can be used to execute various interactive computer software applications (e.g., Microsoft and / or any other type of software) can be used to organize, analyze and / or store data in various (e.g., tabular) formats. Alternatively, computing system 700 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing spreadsheet documents, word processing documents, and / or any other objects, etc.), computing functions, communication functions, etc. The application programs may include various plug-in functions (e.g., provided by SAP of Waldorf, Germany). The functionality may be SAP Integrated Business Planning for Microsoft Excel provided as part of SAP Business Suite, or may be a stand-alone computing product and / or functionality. When activated within an application, the functionality may be used to generate a user interface provided via input / output device 740. The user interface may be generated by computing system 700 and presented to a user (e.g., on a computer screen monitor, etc.).

[0067] One or more aspects or features of the subject matter described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGA) computer hardware, firmware, software and / or combinations thereof. These different aspects or features can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, the programmable processor can be dedicated or general, coupled to receive data and instructions from a storage system, at least one input device and at least one output device, and to send data and instructions to the storage system, at least one input device and at least one output device. A programmable system or computing system can include a client and a server. The client and the server are usually far away from each other, and usually interact through a communication network. The relationship between the client and the server is generated by a computer program running on each computer, and has a client-server relationship between each other.

[0068] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, device and / or apparatus for providing machine instructions and / or data to a programmable processor, such as a disk, an optical disk, a memory and a programmable logic device (PLD), including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily, such as a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium. A machine-readable medium may alternatively or additionally store such machine instructions in a transient manner, such as a processor cache or other random access memory associated with one or more physical processor cores.

[0069] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and a pointing device (e.g., a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive tracking pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0070] In the above description and claims, phrases such as "at least one" or "one or more" may appear after a connected list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which it is used, such phrases are intended to represent any element or feature listed independently, or any listed element or feature combined with any other listed element or feature. For example, the phrases "at least one of A and B", "one or more of A and B;" and "A and / or B" all mean "A alone, B alone, or A and B together". A similar interpretation also applies to lists including three or more items. For example, the phrases "at least one of A, B and C"; "one or more of A, B, C", "A, B and / or C" all mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together". The term "based on" used above and in the claims is intended to mean "based at least in part", so that uncited features or elements are also allowed.

[0071] Depending on the desired configuration, the subject matter described herein may be implemented in a system, device, method and / or article. The embodiments described in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely some examples consistent with aspects related to the described subject matter. Although some changes have been described in detail above, other modifications or additions are also possible. Specifically, in addition to the features and / or variations described herein, features and / or variations may also be provided. For example, the above embodiments may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several additional features disclosed above. In addition, the logic flow depicted in the drawings and / or described herein does not necessarily require the specific order or sequence shown to achieve the desired results. For example, the logic flow may include operations different from and / or additional to those shown without departing from the scope of the present disclosure. One or more operations of the logic flow may be repeated and / or omitted without departing from the scope of the present disclosure. Other embodiments may be within the scope of the appended claims.

Claims

1. A computer-implemented method comprising: scanning, by a first scan engine, first data from a remote memory into a plurality of buffers, each buffer of the plurality of buffers receiving a dedicated portion of the first data; In response to determining that a first buffer of the plurality of buffers is full, loading a first dedicated portion of the first data from the first buffer to a first memory queue, the database system comprising the plurality of buffers and the first memory queue; responsive to determining that a second buffer of the plurality of buffers is full, loading a second dedicated portion of the first data from the second buffer into a second memory queue; in response to determining that the first memory queue is full, transferring, by a first mapping engine, contents of the first memory queue to a first map stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of second data from a second scanning engine, wherein the first data and the second data include mapping data; as well as In response to determining that the second memory queue is full, the second mapping engine transmits the contents of the second memory queue to a second mapping stored in the memory of the database system, wherein the contents of the second memory queue include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine.

2. The computer-implemented method of claim 1, wherein the remote memory is external to the database system, and wherein the first scan engine and the second scan engine are interfaced with the remote memory.

3. The computer-implemented method of claim 1, wherein the mapping data comprises file identifier mapping data and / or large binary object file identifier mapping data. 4 . The computer-implemented method of claim 1 , wherein the first memory queue comprises a single-in, all-out queue.

5. The computer-implemented method of claim 1, wherein the loading of the first dedicated portion of the first data and the loading of the second dedicated portion of the first data are completed in separate parallel threads.

6. The computer-implemented method of claim 1 , further comprising: In response to determining that the second memory queue is full, the first mapping engine transfers the contents of the second memory queue to the first mapping stored in the memory, wherein the contents of the second memory queue include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine.

7. The computer-implemented method of claim 1 , further comprising: receiving, by the database system, a request related to accessing the mapping data from a client device; as well as The request is processed by the database system by accessing at least mapping data in a first mapping stored in the memory.

8. The computer-implemented method of claim 1, wherein the dedicated portion of the first data is based on one or more attributes of the first data.

9. A system comprising: at least one data processor; and at least one memory storing instructions that, when executed by the at least one data processor, result in operations comprising: scanning, by a first scan engine, first data from a remote memory into a plurality of buffers, each buffer of the plurality of buffers receiving a dedicated portion of the first data; In response to determining that a first buffer of the plurality of buffers is full, loading a first dedicated portion of the first data from the first buffer to a first memory queue, the database system comprising the plurality of buffers and the first memory queue; responsive to determining that a second buffer of the plurality of buffers is full, loading a second dedicated portion of the first data from the second buffer into a second memory queue; In response to determining that the first memory queue is full, transferring, by a first mapping engine, contents of the first memory queue to a first map stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of second data from a second scanning engine, wherein the first data and the second data include mapping data; and In response to determining that the second memory queue is full, the second mapping engine transmits the contents of the second memory queue to a second mapping stored in the memory of the database system, wherein the contents of the second memory queue include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine.

10. The system of claim 9, wherein the mapping data comprises file identifier mapping data and / or large binary object file identifier mapping data.

11. The system of claim 9, wherein the first memory queue comprises a single-in, all-out queue.

12. The system of claim 9, wherein the loading of the first dedicated portion of the first data and the loading of the second dedicated portion of the first data are completed in separate parallel threads.

13. The system of claim 9, the operations further comprising: In response to determining that the second memory queue is full, the first mapping engine transfers the contents of the second memory queue to the first mapping stored in the memory, wherein the contents of the second memory queue include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine.

14. The system of claim 9, the operations further comprising: receiving a request from a client device related to access of the mapping data; as well as The request is processed by accessing at least map data in a first map stored in the memory.

15. The system of claim 9, wherein the dedicated portion of the first data is based on one or more attributes of the first data.

16. A non-transitory computer readable storage medium comprising program code which, when executed by at least one data processor, causes operations comprising: scanning, by a first scan engine, first data from a remote memory into a plurality of buffers, each buffer of the plurality of buffers receiving a dedicated portion of the first data; In response to determining that a first buffer of the plurality of buffers is full, loading a first dedicated portion of the first data from the first buffer to a first memory queue, the database system comprising the plurality of buffers and the first memory queue; responsive to determining that a second buffer of the plurality of buffers is full, loading a second dedicated portion of the first data from the second buffer into a second memory queue; in response to determining that the first memory queue is full, transferring, by a first mapping engine, contents of the first memory queue to a first map stored in a memory of the database system, wherein the contents of the first memory queue include a first dedicated portion of the first data from the first scanning engine and at least a first dedicated portion of second data from a second scanning engine, and wherein the first data and the second data include mapping data; as well as In response to determining that the second memory queue is full, the second mapping engine transmits the contents of the second memory queue to a second mapping stored in the memory of the database system, wherein the contents of the second memory queue include a second dedicated portion of the first data from the first scanning engine and at least a second dedicated portion of the second data from the second scanning engine.

17. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: receiving, by the database system, a request related to accessing the mapping data from a client device; as well as The request is processed by the database system by accessing at least mapping data in a first mapping stored in the memory.

18. The non-transitory computer-readable storage medium of claim 16, wherein the dedicated portion of the first data is based on one or more attributes of the first data.

Images