Processing large query results in a database accelerator environment

Abstract

A computer-implemented method for facilitating large scale data transfer from a first data management system to a second data management system is disclosed, the method including receiving data from the first data management system by a first buffer component, dynamically rerouting the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data, forwarding the rerouted data by the second buffer component when the first buffer component is again ready to receive the rerouted data from the second buffer component, and transmitting by a transmitting component the data buffered in the first buffer component to the second data management system.

Description

[Technical Field]

[0001] The present invention relates generally to methods for facilitating large data transfers, and more particularly to a computer-implemented method for facilitating large data transfers from a first data management system to a second data management system. The present invention further relates to a data transfer system for facilitating large data transfers from a first data management system to a second data management system, and an associated computer program product. [Background technology]

[0002] Today, increasingly complex transactional systems, associated data management, very large databases, and separate (or sometimes integrated) OLTP (online transaction processing) / OLAP (online analytical processing) requirements are part of the daily challenges facing enterprise IT (information technology) organizations. In accelerated relational database systems, analytical queries are typically executed on a database accelerator, providing significant performance benefits for analytical query processing. The OLTP portion of the data processing system is left with only a minor impact, allowing it to operate without significant impact from OLAP requirements. Therefore, analytical queries are typically offloaded to the accelerator and forwarded to a dedicated backbone database management system (DBMS) optimized for intensive analytical workloads, and query results are returned to the client application. This offloading must be completely transparent to applications that operate against the OLTP portion of the database. That is, the different behavior of the accelerated DBMS and the backend DBMS must be compensated for, for example, by adapter components that convert resulting data from the backend format to the format required by the client application.

[0003] A typical analytical query result set is intended to be stored and used by a user without additional complex post-processing analytical steps. Accelerator server applications, which act as adapters for back-end databases, transform and transfer data and are optimized for processing such small result sets, which typically contain only a few rows.

[0004] Database operators face a different scenario when executing queries that can generate result sets that must be further analyzed and / or reduced by data consumers, i.e., client applications connecting to the accelerated database. In extreme cases, when tables or large join results are involved in such result sets, the result sets can easily exceed hundreds of gigabytes of data. This scenario can be named the "large result set scenario."

[0005] Unlike typical small result set scenarios, such large result sets require special treatment because a significant amount of processing time and resources that would otherwise be negligible is required for and by the database adapter component. That is, with small result sets, the query execution in the back-end DBMS dominates the entire data processing pipeline. In contrast, with a large query result, the database adapter needs to fetch many rows from the back-end DBMS, transform them into the target format, and send them to the accelerating DBMS (i.e., the OLAP part).

[0006] There are several disclosures related to methods for operating databases and associated accelerators. U.S. Patent Application Publication No. 2012 / 0141794 (A1) discloses a system for enhancing query processing for a relational database. The system includes a server associated with a database containing a plurality of tuples. The server includes processing circuitry for executing a query engine configured to receive a query containing one or more operators. The server is further configured to process the plurality of tuples received from a storage medium storing the database and propagate the operators to a memory management module configured to return each complying tuple that complies with the operator. Furthermore, U.S. Patent Application Publication No. 2016 / 0232206 (A1) discloses that a second query execution party added to a database management system includes a first query execution unit configured to execute the query based on an empirical safety plan for queries against the database and return the query execution results to the query issuer, and a buffer management unit configured to read data for a read request from the database if the data for the read request is not stored in a buffer, store the data in a buffer, and provide the data to the issuer of the read request.

[0007] However, none of these publications address the issues of manual tuning required, buffer overflows, or back-end system outages due to buffer overflows, or a combination thereof, when operating a data management system in conjunction with a back-end system operating as a query accelerator.

[0008] To address this conflict, several techniques have been developed, primarily based on buffering data within the adapter components. However, such intermediate buffering may require advanced knowledge of the data flow from the back-end system to the OLTP part (i.e., the acceleration database) and a large amount of heuristic tuning. This process can be error-prone and result in buffer overflows, or, in the worst case, interruptions or significant delays in the back-end DBMS processing. This situation is unacceptable, and therefore may need to be overcome by self-tuning the adapter components to handle both large and small result set scenarios. Summary of the Invention

[0009] According to one aspect of the present invention, a computer-implemented method for facilitating large-scale data transfer from a first data management system to a second data management system may be provided. The method may include receiving data from the first data management system by a first buffer component and dynamically rerouting the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data. When the first buffer component is again ready to receive the rerouted data from the second buffer component, forwarding the rerouted data by the second buffer component and transmitting the data buffered in the first buffer component to the second data management system by a transmitting component.

[0010] According to another aspect of the present invention, there may be provided a data transfer system for facilitating large-scale data transfer from a first data management system to a second data management system, the system may include a processor and a memory communicatively coupled to the processor, the memory storing program code portions that, when executed, cause the processor to: receive data from the first data management system by a first buffer component; and dynamically reroute the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data.

[0011] When the processor executes the program code portions, it may be further enabled to, when the first buffer component is again ready to receive the rerouted data from the second buffer component, forward the rerouted data by the second buffer component and to transmit by the transmitting component the data buffered in the first buffer component to a second data management system.

[0012] The proposed computer-implemented method for facilitating large scale data transfer from a first data management system to a second data management system may provide many advantages, technical effects, contributions, and / or improvements.

[0013] In general, the proposed method and system can resolve the above conflict by providing a solution to the existing problems of existing spill-to-disk implementations. In particular, the conflict between fast fetching and slow fetching (equivalent to large result set scenarios versus small result set scenarios) can be successfully addressed. Essentially, additional configuration of the adapter between the first and second data management systems can be eliminated. This is due to the superior properties of the proposed solution. Both workload types are handled carefully. The operator does not need to set, reset, or refine / tweak parameter values, or any combination thereof, and a specific mix of workload types is assumed for the first database. There is no need to specify a threshold size or latency value before spilling triggers, or the number of concurrent queries that must be active to indicate resource shortage.

[0014] In interactive application scenarios where large result set queries may occur in conjunction with slow-fetching clients, special precautions may not need to be taken, so the back-end database (i.e., the primary data management system) can pump out results without interruption and without the risk of buffer overflow.

[0015] Due to the large number of possible configuration values, it can be very difficult in practice and can cause a lot of workload and analysis effort by the support engineers responsible for tuning the accelerator installation for customer workloads. With the solution proposed here, such scenarios are a thing of the past, because the global accelerator server configuration can now work immediately for any client query without significant configuration effort. With continuous alternating data paths (depending on the type of result set sent from the first data management system), conversion of query results to client format may always be possible.

[0016] Additionally, discarding result sets after canceling a query can become obsolete, potentially wasting a lot of resources processing rows that will never be fetched (i) to convert result set rows to client format, (ii) to compress the converted rows before they are stored on disk, and (iii) to store the rows in the file system.

[0017] The concept of automatically deciding between a bypass data path (used primarily in small result set scenarios) and a more sophisticated spillover data path (used primarily in large result set scenarios where buffer overflow situations may occur) could build one of the core concepts proposed here. If a spillover data path is not needed, the associated thread is stopped and the buffers on this data path are not needed, so essentially no resource consumption can occur.

[0018] Thus (in short), the concept proposed here clearly supports the reduction of required computing resources as well as the requirement for fine-tuning of configuration parameters, and ultimately eliminates the risk of buffer overflows.

[0019] Below, additional embodiments of the inventive concepts (applicable to methods as well as systems) are described.

[0020] According to a useful embodiment of this method, a first data management system can be optimized for analytical workloads, and according to another useful embodiment of this method, a second data management system can be adapted to receive queries from a software application. Thus, two data management systems (e.g., both implemented as different flavors of database management systems) can each be optimized for a specific task. The second database management system can be optimized to interact with applications in an OLTP manner, while the first database management system can support, or accelerate, the second database management system for OLAP queries. These can be offloaded from the second database management system to the first database management system for execution.

[0021] According to a preferred embodiment of the method, processing the rerouted received data by the second buffer component may include compressing the rerouted received data, storing the compressed rerouted received data, and decompressing the rerouted received data. In this manner, a spillover buffer that can immediately store the rerouted received data may require reduced or negligible storage capacity and, therefore, limited system resources.

[0022] According to an optimized embodiment of this method, compressing, storing, and decompressing the rerouted received data may be performed by one or more processing threads in an interleaved manner on packets of the rerouted received data. Processing may be performed serially on one data block, or processing may be performed in parallel on different data blocks. This may improve overall performance of processing the rerouted data. Because the data volume on this processing branch may be enormous compared to the direct route (or bypass route), it may be beneficial to enable high throughput on the spillover data route to enable the first database management system to send query results with the best possible performance. In this way, large result set scenarios do not present a bottleneck in the coupling between the first and second database systems.

[0023] According to an advantageous embodiment of the method, one or more processing threads of the second buffer component may be idle while not receiving any rerouted receive data from the first buffer component. Thus, when a data path is configured to use a bypass route, resource consumption may not occur in the spillover data path, and setting of heuristic parameters may not be required to customize the selection of a different data path.

[0024] According to another preferred embodiment of the method, the first buffer component can be adapted to convert the received data according to the predefined requirements of the second database management system. This useful step can format the data received from the second database management system as if the query had been made to the second database management system. Thus, the data received from the first database management system can be passed to the application without any additional conversion required. This can help maintain the performance level of the second database management system as high as possible.

[0025] According to a further preferred embodiment of this method, the second data management system can be a row-oriented relational database system. Experience has shown that this type of data management system can be optimized as a data backend for OLTP applications.

[0026] In contrast, according to one alternative embodiment of this method, the first data management system can be a column-oriented database management system. Experience has also shown that this type of data management system can have very good characteristics for OLAP-type requirements. Dedicated data management systems and data organizations that are optimized for analytical workloads can be used, and the data can be organized by columns instead of rows.

[0027] According to an enhanced embodiment, the method may also include determining the data rate of data being transferred from the first buffer component to determine whether the first buffer component is ready to receive rerouted data from the second buffer component again. This strategy may prove superior to simply measuring the fill level of the first buffer component. Measuring the transfer (out) rate of the first buffer component may also help operate the first data management system at a maximum performance level and avoid buffer overflows at the interface between the first data management system and the second data management system.

[0028] According to another advanced embodiment of this method, the sequence (or sorting) of the data transmitted from the first buffer component can correspond to the query that triggered the first buffer component to receive the data from the first data management system. This can be a useful feature, particularly when queries from the second data management system to the first data management system may rely on ordering of query result set data records, i.e., when an SQL statement including "...ORDER BY..." may be used.

[0029] According to a further developed embodiment of the method, the second buffer component may include a data path including at least one component from the group including a compression buffer, a compressor thread, a spilling file write buffer, a spilling file writer thread, a spilling file queue, a spilling file reader thread, a decompression buffer, and a decompressor thread. Thus, to avoid buffer overflow and enable high data throughput from the high performance first data management system, the data path of the second buffer component may enable a flexible method for handling rerouted data.

[0030] Furthermore, embodiments may take the form of an associated computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any apparatus that includes means for storing, communicating, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0031] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and following description that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular any combination between features of a method-type claim and a feature of an apparatus-type claim, is considered to be disclosed herein.

[0032] The above-defined and further aspects of the invention are apparent from and will be elucidated with reference to the example embodiments described hereinafter, which are not limiting of the invention.

[0033] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computer-implemented method of the present invention for facilitating large-scale data transfer from a first data management system to a second data management system. [Figure 2] FIG. 1 is a block diagram of one embodiment of a first data management system connected to a second accelerated data management system. [Figure 3] FIG. 2 is a block diagram of one embodiment of the general functional blocks of an adapter between a first data management system and a second data management system. [Figure 4] FIG. 2 is a block diagram of one embodiment of more detailed functional blocks of a portion of an adapter between a first data management system and a second data management system. [Figure 5] FIG. 1 is a block diagram of one embodiment of the proposed concept of parallel execution data paths (i.e., serial and synchronous result set fetching mechanism with spill-to-disk capability). [Figure 6] FIG. 1 is a block diagram of a FIFO spill file set according to a possible embodiment. [Figure 7] 1 is a block diagram of one embodiment of a data transfer system of the present invention for facilitating large-scale data transfers from a first data management system to a second data management system. [Figure 8] FIG. 8 illustrates an embodiment of a computing system including a system according to FIG. 7. DETAILED DESCRIPTION OF THE INVENTION

[0035] In the context of this description, the following conventions, terms, or expressions, or combinations thereof, may be used.

[0036] The term "first data management system" herein may refer to a tandem accelerating data management system that includes an accelerated data management system (particularly an OLTP database) and a back-end data management system (particularly a database optimized for OLAP workloads).

[0037] The term "second data management system" can consequently refer to an accelerated data management system, i.e., an OLTP database that interfaces with an application. Where OLAP queries can be passed from an application to an OLTP database, the OLAP queries can be routed through an accelerator database, i.e., a dedicated database management system optimized to handle OLAP queries.

[0038] The term "large-scale data transfer" can mean that large amounts of data may have to be transferred from one data management system to another. A typical example could be the transfer of data from an OLAP database, for example, from a query that includes joins between tables, where a potentially unexpected large amount of data may build the result set, rather than the typical small amount of data.

[0039] The term "first buffer component" herein may refer to a storage device adapted to receive data from a backend system, e.g., a first data management system, using a fetcher thread. The buffer may also be referred to as a conversion buffer, since it may intermediately store data received from the first data management system to convert the data into a form required by the requesting component, e.g., a second data management system. When the data flow, i.e., the data path, is based solely on the first buffer component, it may be referred to as a bypass mode.

[0040] The term "second buffer component" can refer to multiple components when the data path does not use bypass mode (as opposed to a simpler form of first buffer component that is simply a storage device). Instead, the fetcher thread does not push received data in bypass mode directly into the translation buffer. Instead, intermediate storage is used. To this end, data may be compressed, transferred to a spilling file write buffer, decompressed, and then simply pushed into the translation buffer. When a second buffer component can be used, the mode of operation may be referred to as "spilling mode."

[0041] The term "predefined fill level" may refer, for example, to the percentage of the conversion buffer that the data can occupy.

[0042] The term "ready to receive rerouted data" may mean that the translation buffer may have enough free space to receive the data. Due to delays caused by the translation thread, the send buffer, and the transmitter thread, the translation buffer may need to have enough buffer capacity to receive a large amount of data at a high rate.

[0043] The term "sending component" may refer to the combination of a converter thread that reads data from a conversion buffer and stores it in a transmission buffer, and a transmitter thread that is adapted to ultimately transmit the data to a second data management system, i.e., an accelerated database management system.

[0044] The figures are described in detail below. All designations in the figures are schematic. First, a block diagram of one embodiment of a computer-implemented method of the present invention for facilitating large-scale data transfer from a first data management system to a second data management system is provided. Thereafter, further embodiments, as well as embodiments of a data transfer system for facilitating large-scale data transfer from a first data management system to a second data management system, are described.

[0045] 1 illustrates a block diagram of a preferred embodiment of a computer-implemented method 100 for facilitating large-scale data transfers from a first data management system to a second data management system. The method includes receiving 102 data from the first data management system by a first buffer component (e.g., including, among other things, a translation buffer and a fetcher thread) and dynamically rerouting 104 the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to buffer and process the rerouted data in a spillover data path. The second buffer component is thereby adapted to process (e.g., compress, store, and decompress) the rerouted received data as data blocks.

[0046] The method 100 further includes, when the first buffer component is again ready to receive the rerouted data from the second buffer component, forwarding the rerouted data by the second buffer component 106, and transmitting the data buffered in the first buffer component to the second data management system by a transmitting component 108. The transmitting component may thereby be assumed to include a converter thread, a transmitting buffer, and a transmitter thread.

[0047] FIG. 2 shows a block diagram of an embodiment 200 of a first data management system connected to a second accelerated data management system. The first data management system is represented here as a database accelerator 212, while the second database management system is depicted as an accelerated database management system 202 including database tables 204. The second DBMS interfaces 207 with one or more applications 206. The database accelerator 212 includes a dedicated database 214 (which may be optimized for OLAP transactions, such as an IBM Netezza product) and a database adapter 218 that sends 220 queries received 208 from the accelerated database management system 202 to the dedicated database 214 including tables 216. The database adapter 218, in turn, receives 222 result sets from the dedicated database 214 and returns 210 the data in the requested format to the accelerated database management system 202, i.e., the second data management system. Note that the term "data management system" is a more general term than a "database management system" that includes a database.

[0048] In other words, queries of the accelerated relational database system 202 are executed on the database accelerator 212 to achieve significant performance benefits for analytical query processing. Therefore, queries are offloaded to the database accelerator 212 and forwarded to a dedicated backend database management system 214 optimized for such analytical workloads, and the query results are returned to the client application 206. This offloading must be completely transparent to the application; that is, the different behavior of the accelerated DBMS and the backend DBMS must be compensated for by, for example, an adapter 218 (or adapter component) that converts result set data from the backend format to the format required by the client application.

[0049] Typical analytical query result sets are intended to be stored and used by consumers (applications) without additional complex post-processing analytical steps. Accelerator server applications, which act as adapters for back-end databases, transform and transfer data and are typically optimized for processing such small result sets containing only a few rows. However, as discussed in the "Background" section, scenarios with large result sets may arise that require special attention and optimized functionality from the database adapter 218.

[0050] FIG. 3 shows a high-level block diagram of one embodiment 300 of the general functional blocks of the database adapter 218 between the first data management system 212 and the second data management system 202 (compare with FIG. 2). The illustrated data pipeline in the application of the database adapter 218 includes spilling logic for all queries, regardless of fetching behavior. This avoids the cumbersome configuration of spill-to-disk heuristics that must be applied by the accelerator server application. It also applies asynchronous processing to the entire spilling logic. This hides access latency when a query begins to spill by overlapping the spilling phase with other processing stages. The spilling logic is then pushed back toward the backend database, so that unconverted rows are spilled. This saves processing resources for unnecessary conversion operations, which accelerates the spilling process because rows can also be saved as files; therefore, backend database resources can be freed up sooner when convergence is applied to spilling.

[0051] In particular, data, i.e., result set rows and backend format, is fetched 302 from the first data management system 212. In the case of large result set scenarios, the data is immediately stored on the local disk 312 in a spill file queue 314 rather than being passed directly to the conversion buffer 304 in bypass mode. The decision to use bypass mode or spill-to-disk mode is made by the fetcher thread (302, fetch). After conversion (306, convert), the result set row has been converted to the required client format and is pushed into a send buffer 308. From here, it is sent 310 to the second data management system 202. The reverse arrow 312 may indicate that the looping process repeats in chunks until all result set rows have been received from the first data management system 212.

[0052] FIG. 4 shows a block diagram of one embodiment 400 of more detailed functional blocks of a portion of the DB adapter (218, see FIG. 2) between the first data management system 212 and the second data management system 202. The fetcher thread 402 can decide to either use bypass mode 430 and send data directly to the translation buffer 420 or activate spill mode. Note that the fetcher thread 402 and translation buffer 420, i.e., the first buffer component, form a close relationship. The fetcher thread 402 then establishes a link between the backend system and the device-like first buffer component.

[0053] However, if the translation buffer 420 is not ready to receive additional data from the fetcher thread 402 (e.g., because a predefined fill rate has been reached or because the data output transfer rate is not high enough), the fetcher thread 402 uses the spilling mode starting at 428. Here, the data path includes the compression buffer 404, the compressor thread 406, the spilling file write buffer 408, the spilling file writer thread 410, the spilling file queue 412, the spilling file reader thread 414, the decompression buffer 416, the decompressor thread 418, and finally the translation buffer 420. In addition, the potentially required sequence of the query result set can also play a determining role for the fetcher thread's 402 decision. If sequencing is required (e.g., because the query includes "...ORDER BY..."), all query result rows of related queries should use the same mode.

[0054] In other words, result set rows are fetched from the backend database (i.e., the first data management system 212) in backend format by fetcher thread 402. In contrast to existing implementations, the rows are not yet converted by the transformer thread but are processed by the spilling component. This avoids implementing spilling heuristics and avoids consuming processing resources converting result set rows if the result set is discarded before it is fully fetched. Rows in backend format are grouped into two blocks in compression buffer 404. Compressor thread 406 receives the blocks, optionally compresses them by applying known compression techniques, such as dictionary encoding, run-length encoding, or heavy-weight zip compression, and transfers the compressed blocks to spill file write buffer 408. From there, the blocks are written to disk-based spill file queue 412 by spill file write data thread 410. The spitting file reader thread 414 deserializes the compressed blocks back into memory and places them in the decompression buffer 416, and the decompressor thread 418 applies the corresponding decompression algorithm to store the rows in the backend format.

[0055] The block of rows is then converted by a converter thread 422 into the required client format, and a transmitter thread 426 transmits it from a send buffer 424 to the client application (cf. FIG. 2, 206) via the database accelerator network connection and through a second data management system (cf. FIG. 2, 202).

[0056] Figure 5 shows a block diagram of one embodiment of the proposed concept of a parallel execution data path 500 (i.e., a serial asynchronous result set fetch mechanism with spill-to-disk capability). The serial fetching time represented by the first row of blocks 502a, ..., 506c progresses in time from right to left. The serial formatting includes the sequence: fetch B1 (block 1 of data), transform B1, send B1, fetch B2, transform B2, send B2, fetch B3, transform B3, and send B3.

[0057] However, in asynchronous fetching with spill-to-disk, blocks B1, B2, and B3 are processed in parallel and interleaved. The first row of blocks 502, representing fetch B1, fetch B2, and fetch B3, is executed without delay by fetcher thread 402 (contrast with Figure 4). Subsequently, compressor thread 406, spilling file writer thread 410, spilling file reader thread 414, decompressor thread 418, transformer thread 422, and sender thread 426 process the blocks as shown: compress 508B1, 508a; compress B1, 508b; compress B1, 508c; serialize B1, 510a; serialize B1, 510b. deserialize B1 512a, deserialize B2 512b, deserialize B3 512c; decompress B1 514a, decompress B2 514b, decompress B3 514c; transform B1 504a, transform B2 504b, transform B3 504c; and send B1 506a, send B2 506b, send B3 506c.

[0058] Therefore, this diagram shows overlapped processing phases and how parallel execution phases (i.e., in spill mode) hide access latency when rows are fetched from the client compared to serial execution (i.e., in bypass mode). Note that in the serial and asynchronous cases there is a ramp-up phase between the sending of block 1 and block 2. This latency can be avoided if the bypass optimization is applied. The enhanced processing pipeline only requires as parameters the buffer size and the compression algorithm to be applied. Both values ​​can be determined much more easily than in current conventional configurations.

[0059] In what follows, several special cases should be considered: ordered query results, queries without specified result set ordering, result set compression, and spill file organization.

[0060] Ordered query results: Writing result set rows to spill files is only necessary if both of the following conditions are true: (1) the result set is larger than the (pre-allocated) internal memory buffer to process it, and (2) the client application is fetching rows at a slower rate than they are being produced by the accelerator.

[0061] Therefore, as shown in Figure 5, the fetcher thread can skip the spilling portion of the pipeline if the transformation buffer is not full. That is, when the pipeline starts, fetched result set rows can be written directly to the transformation buffer, allowing the transformer thread to begin work immediately. This avoids the initial access latency during the ramp-up phase of the pipeline (contrast Figure 5).

[0062] If the conversion buffer overflows (which will happen if the client application is fetching slowly and the result set is large enough), spilling can begin so that the result set rows are drained from the backend database as fast as possible. The asynchronous processing discussed earlier means that spilling is performed as a background operation, ensuring that, for example, in the use case of interactive, paged result set browsing, the conversion buffer is filled as soon as the next chunk of rows is fetched by the client.

[0063] If the hardware is sized appropriately—that is, the disk I / O bandwidth is high enough for writing blocks to and reading blocks from the spill files—and there are sufficient CPU resources to compress the data at high throughput—the spilling overhead will not be observable from the client's perspective. The hardware configuration can be determined during development time because it does not depend on the client's workload. Therefore, workload-dependent heuristics, such as the number of concurrent query tasks or the delay before triggering spilling, do not need to be configured.

[0064] The buffer size can be adjusted to control which queries are eligible for spilling. In practice, a few megabytes is sufficient in real-world scenarios, and the pre-set default value is adequate for most use cases. To adjust the behavior for a specific client, the size can be easily increased by determining the maximum result set size, which is a query-specific characteristic and does not depend on the scheduling order of multiple queries.

[0065] Query with no specified result set ordering: Note that in general, the asynchronous fetching pipeline must maintain the order of result blocks. That is, when a block is written to the spill file, it must be converted before any subsequently fetched blocks. Therefore, bypass mode can only be used when the spill file is empty and in the initialization phase, or when all previously spilled blocks have already been exhausted.

[0066] In the special case where a query does not require a particular ordering of result set rows, i.e., when no ORDER BY clause is present in the query statement, additional optimizations can be applied. This can be decided at query initiation time, or asynchronously while the query is running, allowing optimizations to be made once the analytical results are ready.

[0067] If no ordering constraint is specified, the database system is free to return result set rows in any order. This can be exploited to trigger bypass mode as soon as there is space available in the transformation buffer. In this case, the fetcher thread can write directly to the transformation buffer without having to synchronize with previously spilled blocks. This optimization is particularly useful when the spilling pipeline processes rows at a slower rate than the client can fetch them due to hardware constraints, for example, when available processing and I / O bandwidth is limited by many parallel queries.

[0068] Result Set Compression: Compressing result set data before writing it to spill files is an optional behavior that can be configured via global accelerator server parameters. If compression is not used, the compressor (decompressor) thread simply forwards blocks to subsequent stages without modifying them. When compression is enabled, additional CPU resources are expended to reduce I / O time and file system usage.

[0069] Algorithms that achieve a higher compression level require more CPU resources to compress (decompress) blocks of result set rows, but can improve the scalability of the overall accelerator system because more data can be stored on the local disk. Therefore, the trade-offs that can be made must be determined by calibrating the accelerator before deploying it in a production environment. By default, a medium compression level should be chosen, which results in reduced disk space and fast enough compression (decompression) throughput so that the transformation buffer can be filled at a faster rate than it is consumed.

[0070] Spill file organization: For typical query processing use cases that require result set ordering, the spill file queue should be organized as a disk-based FIFO (first-in, first-out) data structure. File systems typically do not natively support this behavior, which requires reading and freeing data blocks on the consumer side and allocating and writing data blocks on the producer side.

[0071] To solve this, for example, several implementation techniques can be applied: (i) the result set may be spilled to an append-only file that is released once the entire result set is consumed, (ii) the result set may be spilled to a random access file, and the FIFO behavior is modeled by an in-memory index structure, i.e., the location of a particular block within the spill file is tracked so that it can be directly read / overwritten by an offset calculation, or (iii) the result set may be spilled to a set of append-only files that are internally managed by a FIFO access pattern.

[0072] Append-only files are simple to implement and offer the lowest management overhead because blocks can be added to the end of the file and consumed from the beginning. The read-write offsets must be synchronized by the spill file reader and writer threads. A FIFO access pattern applied to the entire file results in sequential I / O behavior that offers the best I / O throughput for most disk devices. However, disk space can only be reclaimed after the entire file has been consumed.

[0073] Random access files require detailed tracking of each result set block to determine where it is stored. This can incur the highest management overhead due to the indirect lookup of each block address. Random access patterns are also generally only supported by some disk technologies, such as solid-state disks. In most cases, sequential access behavior results in higher I / O performance. Furthermore, random block access requires all blocks to be of equal size, which is not guaranteed when compression algorithms are applied to reduce block size. In other cases, an additional free space map must be maintained to find the appropriate location for blocks that should be spilled by overwriting previously read blocks.

[0074] Set-based spill file management (contrast with Figure 6) represents the best solution for most hardware configurations. Spill files are divided into a set of append-only spill files with a fixed maximum size. Newly spilled blocks are appended to the last file open for writing, unless capacity has been reached. When a block needs to be read, the corresponding spill file switches to read-only mode, and blocks can be consumed one after another until all blocks have been read. Once all blocks are consumed, the entire file can be released to disk before the result set is completely fetched. This spill file implementation further simplifies synchronization between reader and writer threads, since each file in the set can be processed independently. Further optimizations can be applied when blocks are consumed faster than they can be written, for example, when a client that was previously fetching slowly now fetches rapidly because the next batch of data is requested. If all spill files are exhausted, the reader can steal the currently written spill file before it reaches its capacity. This force-flushing ends the spill phase and allows bypass mode to resume.

[0075] Thus, FIG. 6 shows a block diagram 600 of a FIFO spill file set according to a possible implementation. In the spill file queue 602, spill file blocks are ordered according to block age. Also, a read sequence 618 and a write sequence 620 are symbolically shown. The currently read spill file r 604 contains blocks Br1, ..., Brn. From here, the next block is read, and the file is released after all blocks have been consumed by the spilling file reader thread 614 (compare with FIG. 4, 414). Spill files 606 (with blocks B11, ..., B1n) through 608 (with blocks Bi1, ..., Bin) each represent file blocks 624 ready to be read, each with a capacity of n blocks.

[0076] The currently written spill file w 612 contains blocks Bw1, ..., Bwx. When the conversion buffer capacity is reached by the spilling file writer thread 616 (see also 408, Figure 4), the next block is added and allocated to a new file. Connection 622 indicates that the last file is stolen when the previous file is exhausted and bypass mode is triggered.

[0077] 7 illustrates a block diagram of one embodiment of a data transfer system 700 of the present invention for facilitating large-scale data transfer from a first data management system to a second data management system. The system includes a processor 702 and a memory 704 communicatively coupled to the processor, the memory storing program code portions that, when executed, cause the processor to receive data from the first data management system via a first buffer component 708 (particularly via a receiver module 706) and, when the first buffer component 708 reaches a predefined fill level, dynamically reroute the received data to a second buffer component 710, the second buffer component 710 being adapted to process the rerouted received data.

[0078] Additionally, processor 702 executes the stored program code portions to enable, when the first buffer component is again ready to receive the rerouted data from the second buffer component, to forward the rerouted data by the second buffer component and to enable transmission component 712 to transmit the data buffered in first buffer component 708 to a second data management system (corresponding to 420 in FIG. 4).

[0079] It should also be noted that all functional units, modules, and blocks (particularly the processor 702, memory 704, receiver unit 706, first buffer component 708, second buffer component 710, and transmitting component 712) may be communicatively coupled to one another for signal or message exchange in a selected 1:1 manner. Alternatively, the functional units, modules, and blocks may be linked to a system internal bus system 714 for selective signal or message exchange.

[0080] Embodiments of the present invention may be practiced with virtually any type of computer platform, regardless of whether the platform is suitable for storing and / or executing program code. Figure 8 illustrates, by way of example, a computing system 800 suitable for executing program code associated with the proposed method.

[0081] Computing system 800 is merely one example of a suitable computer system, and whether computer system 800 is capable of implementing and / or performing any of the functions described above is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. Computer system 800 has components operable in numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 800 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computer system / server 800 may be described in the general context of computer system-executable instructions, such as program modules, executed by computer system 800. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 800 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0082] As shown in the figure, computer system / server 800 is embodied in the form of a general-purpose computing device. Components of computer system / server 800 may include, but are not limited to, one or more processors or processing units 802, a system memory 804, and a bus 806 that couples various system components, including the system memory 804, to the processor 802. Bus 806 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. Computer system / server 800 typically includes a variety of computer system-readable media. Such media can be any available media that is accessible by computer system / server 800 and includes both volatile and non-volatile media, removable and non-removable media.

[0083] System memory 804 may include computer-system-readable media in the form of volatile memory, such as random access memory (RAM) 808 and / or cache memory 810. Computer system / server 800 may further include other removable / non-removable, volatile / non-volatile computer-system storage media. By way of example only, storage system 812 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided for reading from or writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such cases, each may be connected to bus 806 by one or more data media interfaces. As further illustrated and described below, memory 804 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0084] A program / utility having a set (at least one) of program modules 816 may be stored in memory 804, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 816 typically perform the functions and / or methods of embodiments of the present invention described herein.

[0085] Computer system / server 800 may also communicate with one or more external devices 818, such as a keyboard, pointing device, display 820, etc., one or more devices that allow a user to interact with computer system / server 800, or any device (e.g., network card, modem, etc.) that allows computer system / server 800 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via input / output (I / O) interface 814. Additionally, computer system / server 800 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via network adapter 822. As shown, network adapter 822 may communicate with other components of computer system / server 800 via bus 806. It should be understood that other hardware and / or software components, not shown, may be used with computer system / server 800. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0086] Additionally, a data transfer system 700 for facilitating large scale data transfer from a first data management system to a second data management system can be attached to the bus system 806 .

[0087] The description of various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been chosen to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0088] The present invention may be embodied as a system, method, and / or computer program product, which may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0089] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system for propagation. Examples of computer-readable media can include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read / write (CD-R / W), DVD, and Blu-ray disk.

[0090] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded on them, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0092] Computer-readable program instructions for carrying out operations of the present invention can be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code, including object-oriented programming languages ​​such as Smalltalk®, C++, and the like, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0093] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored constitutes an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0095] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0096] The flowcharts and / or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware instructions and computer instructions.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0098] Structure, material, acts, and equivalents corresponding to all means or steps plus functional elements in the following claims are intended to include any structure, material, or acts for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications as suited to the particular uses contemplated.

[0099] Briefly, the inventive concept can be summarized by the following clauses. (1) A computer-implemented method for facilitating large scale data transfer from a first data management system to a second data management system, comprising: - receiving data from a first data management system by a first buffer component; - dynamically rerouting received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted data from the second buffer component, forwarding the rerouted data by the second buffer component; transmitting the data buffered in the first buffer component to a second data management system by a transmitting component; 20. A computer-implemented method comprising: (2) The method of clause 1, wherein the first data management system is optimized for analytical workloads. (3) The method of clause 1 or 2, wherein the second data management system is adapted to receive queries from the software application. (4) The method of any of clauses 1 to 3, wherein processing the rerouted received data by the second buffer component includes at least one activity selected from the group consisting of compressing the rerouted received data, storing the compressed rerouted received data, and decompressing the rerouted received data. (5) The method of clause 4, wherein compressing, storing, and decompressing the rerouted received data are performed by one or more processing threads in an interleaved manner on packets of the rerouted received data. (6) The method of clause 5, wherein one or more processing threads of the second buffer component are idle while not receiving any rerouted receive data from the first buffer component. (7) The method of any of clauses 1 to 6, wherein the first buffer component is adapted to transform the received data according to predefined requirements of the second data management system. (8) The method of any one of clauses 1 to 7, wherein the second data management system is a row-oriented relational database system. (9) The method of any one of clauses 1 to 8, wherein the first data management system is a column-oriented database management system. (10) determining a data rate of data being transferred from the first buffer component to determine whether the first buffer component is again ready to receive the rerouted data from the second buffer component; 10. The method of any of clauses 1 to 9, further comprising: (11) A method according to any of clauses 1 to 10, wherein the sequence of data transferred from the first buffer component conforms to a query that triggered the first buffer component to receive data from the first data management system. (12) The method of any of clauses 1 to 11, wherein the second buffer component comprises a data path including at least one component from the group including a compression buffer, a compressor thread, a spilling file write buffer, a spilling file writer thread, a spilling file queue, a spilling file reader thread, a decompression buffer, and a decompressor thread. (13) A data transfer system for facilitating large-scale data transfer from a first data management system to a second data management system, comprising: a processor; and a memory communicatively coupled to the processor. Including, The memory stores program code portions that, when executed, cause a processor to: - receiving data from a first data management system by an adapted first buffer component; - dynamically rerouting received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted data from the second buffer component, forwarding the rerouted data by the second buffer component; transmitting the data buffered in the first buffer component to a second data management system by a transmitting component; A data transfer system that can perform the above. (14) The system of clause 13, wherein the first data management system is optimized for analytical workloads. (15) The system of clause 13 or 14, wherein the second data management system is adapted to receive queries from the software application. (16) The system of any of clauses 13 to 15, wherein processing the rerouted received data by the second buffer component includes at least one activity selected from the group consisting of compressing the rerouted received data, storing the compressed rerouted received data, and decompressing the rerouted received data. (17) The method of clause 16, wherein compressing, storing, and decompressing the rerouted received data are performed by one or more processing threads in an interleaved manner on packets of the rerouted received data. (18) The system of clause 17, wherein one or more processing threads of the second buffer component are idle while not receiving any rerouted receive data from the first buffer component. (19) The system of any of clauses 13 to 18, wherein the first buffer component is adapted to convert the received data according to predefined requirements of the second data management system. (20) The system of any of clauses 13 to 19, wherein the second data management system is a row-oriented relational database system. (21) The system of any of clauses 13 to 20, wherein the first data management system is a column-oriented database management system. (22) The processor also executes the program code. determining a data rate of data being transferred from the first buffer component to determine whether the first buffer component is ready to receive the rerouted data from the second buffer component again; 22. The system according to any of clauses 13 to 21, wherein (23) A system as described in any of clauses 13 to 22, wherein the sequence of data transferred from the first buffer component conforms to a query that triggered the first buffer component to receive data from the first data management system. (24) The system of any of clauses 13 to 23, wherein the second buffer component comprises a data path including at least one component from the group including a compression buffer, a compressor thread, a spilling file write buffer, a spilling file writer thread, a spilling file queue, a spilling file reader thread, a decompression buffer, and a decompressor thread. (25) A computer program product for facilitating large scale data transfer from a first data management system to a second data management system, the computer program product including a computer-readable storage medium having program instructions embodied therein, the program instructions being readable by one or more computing systems. - receiving data from a first data management system by a first buffer component; - dynamically rerouting received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted data from the second buffer component, forwarding the rerouted data by the second buffer component; transmitting the data buffered in the first buffer component to a second data management system by a transmitting component; a computer program product executable by said one or more computing systems or controllers to cause said one or more computing systems or controllers to perform

Claims

1. 1. A computer-implemented method for facilitating large scale data transfer from a first data management system to a second data management system, comprising: receiving, by a first buffer component, received data from said first data management system; - dynamically rerouting the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted receive data from the second buffer component, forwarding the rerouted receive data by the second buffer component; transmitting, by a transmitting component, said received data buffered in said first buffer component to said second data management system; 20. A computer-implemented method comprising:

2. The method of claim 1 , wherein the second data management system is adapted to receive queries from a software application.

3. 2. The method of claim 1, wherein said processing of the rerouted received data by the second buffer component includes at least one activity selected from the group consisting of compressing the rerouted received data, storing the compressed rerouted received data, and decompressing the rerouted received data.

4. 4. The method of claim 3, wherein the compressing, storing, and decompressing the rerouted received data are performed by one or more processing threads in an interleaved manner on packets of the rerouted received data.

5. 5. The method of claim 4, wherein the one or more processing threads of the second buffer component are idle while not receiving any rerouted receive data from the first buffer component.

6. The method of claim 1 , wherein the first buffer component is adapted to transform the received data according to predefined requirements of the second data management system.

7. 2. The method of claim 1, wherein the second data management system is a row-oriented relational database system.

8. The method of claim 1 , wherein the first data management system is a column-oriented database management system.

9. determining a data rate of received data being transferred from the first buffer component to determine whether the first buffer component is ready to receive the rerouted received data from the second buffer component again; The method of claim 1 further comprising:

10. 2. The method of claim 1, wherein a sequence of received data transferred from the first buffer component conforms to a query that triggered the first buffer component's receipt of the received data from the first data management system.

11. 2. The method of claim 1, wherein the second buffer component comprises a data path including at least one component from the group including a compression buffer, a compressor thread, a spilling file write buffer, a spilling file writer thread, a spilling file queue, a spilling file reader thread, a decompression buffer, and a decompressor thread.

12. 1. A data transfer system for facilitating large scale data transfer from a first data management system to a second data management system, comprising: a processor and a memory communicatively coupled to said processor; the memory storing program code portions, the program code portions, when executed, causing the processor to: receiving, by a first buffer component, received data from said first data management system; - dynamically rerouting the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted receive data from the second buffer component, forwarding the rerouted receive data by the second buffer component; transmitting, by a transmitting component, said received data buffered in said first buffer component to said second data management system; A data transfer system that can perform the above.

13. The system of claim 12 , wherein the second data management system is adapted to receive queries from a software application.

14. 13. The system of claim 12, wherein said processing of said rerouted received data by said second buffer component includes at least one activity selected from the group consisting of compressing said rerouted received data, storing said compressed rerouted received data, and decompressing said rerouted received data.

15. 15. The system of claim 14, wherein the compressing, storing, and decompressing the rerouted received data are performed by one or more processing threads in an interleaved manner on packets of the rerouted received data.

16. 16. The system of claim 15, wherein the one or more processing threads of the second buffer component are idle while not receiving any rerouted receive data from the first buffer component.

17. The system of claim 12 , wherein the first buffer component is adapted to transform the received data according to predefined requirements of the second data management system.

18. 13. The system of claim 12, wherein the second data management system is a row-oriented relational database system.

19. The system of claim 12 , wherein the first data management system is a column-oriented database management system.

20. The processor also, when executing the program code, determining a data rate of received data being transferred from the first buffer component to determine whether the first buffer component is ready to receive the rerouted received data from the second buffer component again; The system of claim 12 , wherein:

21. 13. The system of claim 12, wherein a sequence of received data transferred from the first buffer component conforms to a query that triggered the first buffer component's receipt of the received data from the first data management system.

22. 13. The system of claim 12, wherein the second buffer component comprises a data path including at least one component from the group including a compression buffer, a compressor thread, a spilling file write buffer, a spilling file writer thread, a spilling file queue, a spilling file reader thread, a decompression buffer, and a decompressor thread.

23. 1. A computer program for facilitating large scale data transfer from a first data management system to a second data management system, the computer program comprising: receiving, by a first buffer component, received data from said first data management system; - dynamically rerouting the received data to a second buffer component when the first buffer component reaches a predefined fill level, the second buffer component being adapted to process the rerouted received data; - when the first buffer component is again ready to receive the rerouted receive data from the second buffer component, forwarding the rerouted receive data by the second buffer component; transmitting, by a transmitting component, said received data buffered in said first buffer component to said second data management system; A computer program that performs the following:

Images