Method and system of ingesting data from a source application

Abstract

A system for ingesting data from a source application is provided. The system includes a write manager (106, 204) and an array manager (108, 206, 308). The write manager receives input data tuples from one or more writer threads (104A-N) of the source application. The array manager identifies a first data array (110A) amongst one or more data arrays for a received first input data tuple from a first writer thread of the one or more writer threads. The array manager inserts the received first input data tuple from the first writer thread into the identified first data array. The array manager identifies a second data array (HOB) amongst the one or more data arrays for a received second input data tuple from a second writer thread of the one or more writer threads. The array manager inserts the received second input data tuple into the identified second data array.

Description

[0001] METHOD AND SYSTEM OF INGESTING DATA FROM A SOURCE APPLICATION

[0002] TECHNICAL FIELD

[0003] The disclosure generally relates to data processing in large-scale parallel applications, and more particularly, the disclosure relates to a method and a system of ingesting data from a source application.

[0004] BACKGROUND

[0005] Modem large-scale applications generate vast amounts of data that require processing to derive or extract meaningful insights. These modem large-scale applications often encompass diverse domains, including large traffic simulations, which simulate the movement of vehicles, pedestrians, and other elements in a road network. For example, a large traffic simulation continuously generates data over time, offering information and insights into various aspects of the simulated environment, such as the behavior of vehicles, traffic lights, road signs, and pedestrians. Despite the high volume of information, it often lacks the specificity required for actionable insights. For example, merely knowing the spatial coordinates (x, y) of vehicles at any given moment may not suffice to understand overall traffic dynamics.

[0006] To extract actionable or meaningful insights from this data, the data requires processing, involving the computation of metrics relevant to domain experts. These metrics may include, for example, the average speed of vehicles on critical roads, or the locations and sizes of traffic jams. By computing these metrics, experts can interpret traffic dynamics comprehensively, including the movement of vehicles, changes in traffic flow, congestion patterns, and overall behavior.

[0007] Similarly, other software systems such as database systems, performance monitoring systems, and logging systems may follow analogous principles. These systems also generate vast amounts of data, requiring analytical processing to extract meaningfuVvaluable insights before they can be considered fully operational or ready for use. However, performing such analytical processing on the data poses significant challenges due to the sheer volume and complexity of the data involved.

[0008] Moreover, software tools like Excel or MATLAB are not suitable for processing and handling vast amounts of data due to their limitations in scale and performance. The traditional post-processing paradigm involves collecting vast amounts of data on a disk and subsequently performing analytical processing tasks on it to derive valuable insights. However, this approach fails to perform real-time analytical processing on the data while it is being generated by the software system.

[0009] Stream processing emerges as a viable alternative, leveraging specialized tools like data stream processing engines. These data stream processing engines, including ingestion systems, data processing engines, and output systems, streamline the processing pipeline. Nonetheless, transitioning data between pipeline stages can introduce complexities, especially when multiple actors or threads are involved. Moreover, maintaining guarantees such as preserving data order poses computational challenges, particularly in high-concurrency scenarios.

[0010] In contrast, some large-scale parallel applications process vast amounts of data in distinct phases and adopt a phased processing approach, dividing tasks into distinct phases for efficient processing. This approach enables concurrent execution of tasks by multiple threads, generating output data consisting of tuples that are logically concurrent. Thus, the order in which these tuples are processed does not require strict ordering in the ingestion system. In other words, the tuples are not required to be processed strictly in the order they are received.

[0011] For example, when updating the simulation data of vehicles from one simulated time (tO) to the next (tl), each simulated vehicle's state needs to be updated accordingly. The simulation data may include the position and speed of each vehicle. In a multi-threaded update computation, the order in which a particular vehicle (A) is updated relative to another vehicle (B) is insignificant for analyzing the simulation's behavior at a given simulated time. In such cases, applying strict order and synchronization can delay the performance of the large-scale parallel application, leading to complexities in managing vast amounts of data and complicating the analysis of simulation behavior at specific points in time.

[0012] Despite advancements, managing concurrent data access remains a critical concern. Existing systems employ lock-based approaches or lock-free algorithms to handle concurrent reading and writing of tuples. The lock-based approaches utilize locks to manage access to internal data structures, which means that only one thread can access the data at a time. The lock-free algorithms manage concurrency at a more granular level without relying on locks. While both approaches enable concurrent access, they face scalability constraints under high contention scenarios, where many threads concurrently access the data for extended periods. For example, the lock-based approaches could face scalability constraints beyond 2 to 3 concurrent threads, while the lock-free approaches could face scalability constraints at approximately 8 to 10 threads.

[0013] Existing solutions like Nebula Stream and Apache Kafka offer alternative approaches to managing data concurrency. Nebula Stream utilizes non-blocking queues to enable multi-threaded access, while Apache Kafka partitions data to facilitate parallel processing. However, these approaches have limitations as they impose a workaround on users and may not fully address the parallelism challenges within individual ingestion systems. Additionally, they introduce complexities, especially in scenarios where data needs to be logically grouped. For example, in tracking website purchases, partitioning orders based on price can lead to difficulties in distinguishing orders containing similar products but differing quantities. This arises because partitions operate autonomously, potentially complicating the analysis of data.

[0014] Therefore, there arises a need to address the aforementioned technical problems / drawbacks of processing data from a source application.

[0015] SUMMARY

[0016] It is an object of the disclosure to provide a method and a system of ingesting data from a source application while avoiding one or more disadvantages of prior art approaches.

[0017] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0018] According to a first aspect, a method of ingesting data from a source application is provided. The method includes receiving input data tuples from one or more writer threads of the source application. The method includes identifying a first data array amongst one or more data arrays for a received first input data tuple from a first writer thread of the one or more writer threads. The identified first data array corresponds to the first writer thread of the source application. The method includes inserting the received first input data tuple from the first writer thread into the identified first data array. The method includes identifying a second data array amongst the one or more data arrays for a received second input data tuple from a second writer thread of the one or more writer threads. The identified second data array corresponds to the second writer thread of the source application. The method includes inserting the received second input data tuple from the second writer thread into the identified second data array.

[0019] The method eliminates the need for synchronization among the one or more writer threads during data ingestion. This elimination enhances the performance of high-performance applications by eliminating the necessity for synchronization between the one or more writer threads, thereby improving ingestion performance. The method is utilized for implementing the ingestion of large-scale traffic simulation data in a product based on the CityMoS application as it effectively meets the demands of stakeholders by ensuring efficient data handling and processing.

[0020] Preferably, a size of each of the first and second data arrays is updated after the received first and second input data tuples are inserted into the first and second data arrays, respectively.

[0021] Preferably, a writer thread identifier is used to identify the first and second writer threads. Preferably, the writer thread identifier is a numeric identifier. Preferably, each of the plurality of data arrays is identified using an identifier with a numeric value corresponding to a value of the writer thread identifier.

[0022] Preferably, a consuming application having a plurality of reader threads reads data from the plurality of data arrays. Preferably, the consuming application initiates a read operation from the plurality of data arrays by requesting a read handle. Preferably, the read handle is initialized by a first reader thread of the consuming application. Preferably, the read handle initialized by the first reader thread contains a reference to a specific data array of the plurality of data arrays.

[0023] Preferably, the read handle initialized by the first reader thread contains two indices, where a first index of the two indices corresponds to a position in the specific data array corresponding to a next data tuple to be read, and a second index of the two indices corresponds to a position in the specific data array which indicates that a next data array should be read from. Preferably, the first reader thread of the consuming application reads a data tuple. The data tuple that is read is identified by the data array and the first index of the read handle.

[0024] Preferably, after reading the data tuple, the first reader thread of the consuming application updates the read handle by increasing the first index by a value of one. Preferably, if the first index is equal to the second index, the read handle is updated with the next data array.

[0025] The method allows the one or more reader threads to independently access the system during data consumption without requiring centralized synchronization. Furthermore, the method eliminates the need for synchronization among the one or more reader threads, resulting in improved analytics computation performance.

[0026] According to a second aspect, a system includes means adapted for carrying out all the steps of the method provided.

[0027] According to a third aspect, a computer program includes instructions for carrying out all the steps of the above method when the computer program is executed on a computer system.

[0028] These and other aspects of the disclosure will be apparent from the implementations) described below.

[0029] BRIEF DESCRIPTION OF DRAWINGS

[0030] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0031] FIG. 1 illustrates a block diagram of a system of ingesting data from a source application in accordance with an implementation of the disclosure;

[0032] FIG. 2 illustrates an operational flow of a system for ingesting data from a source application in accordance with an implementation of the disclosure; FIG. 3 illustrates an operational flow of a system for reading data by a consuming application in accordance with an implementation of the disclosure;

[0033] FIGS. 4A-4C illustrate exemplary diagrams of different stages of data arrays during a write operation in accordance with an implementation of the disclosure;

[0034] FIGS. 5A and 5B illustrate an exemplary diagram of a read operation initiated by a consuming application in accordance with an implementation of the disclosure;

[0035] FIG. 6 is a flow diagram that illustrates a method of ingesting data from a source application in accordance with an implementation of the disclosure; and

[0036] FIG. 7 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented.

[0037] DETAILED DESCRIPTION OF THE DRAWINGS

[0038] Implementations of the disclosure provide a method and a system for ingesting data from a source application.

[0039] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0040] Terms such as “a first”, “a second”, “a third”, and “a fourth” (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms “include” and “have” and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0041] Definitions:

[0042] Thread: A thread is a program that comprises a sequence of computer instructions for achieving a certain task (e.g. changing a flat tire on a car). For example, the process includes (i) finding a safe place to stop, (ii) lifting the car with a tool, (iii) removing the punctured tire, (iv) inserting and tightening a new tire, and (iv) lowering the carback to the ground.

[0043] Now, if changing two tires is necessary, steps (iii) and (iv) are repeated sequentially. Alternatively, if another passenger is available, they can independently change the other punctured tire while the driver completes the first tire replacement, executing steps (iii) and (iv) individually, thereby reducing the time needed to complete the repair. Similarly, in a computer program, threads enable parallel execution of tasks. In other words, the threads are essentially streams of computing operations that run concurrently and independently, allowing for efficient utilization of computing resources and faster completion of tasks.

[0044] Tuple: A tuple is commonly defined as an individual row or record within a table, encapsulating a set of related attributes. Each tuple comprises multiple fields or columns, each corresponding to a specific data element or characteristic. Conceptually, it can be envisioned as a horizontal slice of data within a table, where each field holds a value representing a particular attribute or property. For example, in the context of an e-commerce website's database table, a tuple could represent the details of a single product. It typically includes fields such as a unique numeric product identifier, product name, description, price, and availability status. Each tuple within the table contains distinct values corresponding to the attributes of individual products, facilitating organized storage and retrieval of product information.

[0045] Data Array: A data array is a structured collection of data elements (e.g., value or variables), organized in an order and stored contiguously in the memory of a system.

[0046] FIG. 1 illustrates a block diagram of a system 100 of ingesting data from a source application 102 in accordance with an implementation of the disclosure. The system 100 includes a write manager 106 and an array manager 108. The source application 102 (e.g., a parallel data generator) includes a first writer thread 104A, a second writer thread 104B, and a Nthwriter thread 104N. Preferably, the system 100 is communicatively connected to the source application 102. The source application 102 may be a data producer. Each writer thread (e.g., the first writer thread 104A, the second writer thread 104B, and the Nlwriter thread 104N) includes data tuples of the source application 102 with a writer thread identifier. The writer thread identifier may be a numeric identifier ranging from 0 to n-1. The source application 102 assigns the writer thread identifier to each writer thread to distinguish each writer thread from other writer threads. The writer thread identifier may be used to identify the first writer thread 104A and the second writer thread 104B. This implies that the writer thread identifier differentiates between the initial writer thread and subsequent writer threads. Each writer thread collects the data tuples from one or more external sources using the source application 102. The data tuples may include a first data tuple, a second data tuple, and an nth data tuple. The source application 102 generates the data tuples by simulating events (e.g., activities like traffic simulations) within the source application 102. The array manager 108 includes a first data array 110A, a second data array HOB, and an nth data array HON. Each data array (e.g., the first data array H0A, the second data array HOB, and the nth data array HON) includes a numeric identifier ranging from 0 to n-1.

[0047] The write manager 106 receives the first data tuple from the first writer thread 104A and the second data tuple from the second writer thread 104B of the source application 102. The write manager 106 transmits the first data tuple received from the first writer thread 104A to the array manager 108. The array manager 108 identifies the first data array H0A among the data arrays H0A-N that corresponds to the first writer thread 104A of the source application 102. The array manager 108 identifies the first data array 110 A corresponds to the first writer thread 104A among the data arrays H0A-N using the writer thread identifier of the first writer thread 104A. Preferably, the array manager 108 identifies the first data array H0A corresponds to the first writer thread 104A among the data arrays H0A-N based on a size of each data array H0A-N and a location of each data array in memory of the system 100. Each data array is identified using an identifier that corresponds to each data array. The identifier may include a numeric value corresponding to the value of the writer thread identifier associated with it. The system 100 inserts the received first input data tuple from the first writer thread 104A into the identified first data array 110A. During the insertion process, the mapping of the data arrays H0A-N remains unaffected by an order of the data tuples while the writer threads 104A-N are active. For example, if the third data array2 HOC is identified for the first data tuple, the first data tuple is inserted in the third data array2 HOC.

[0048] The write manager 106 transmits the second data tuple from the second writer thread 104B to the array manager 108. The array manager 108 identifies the second data array HOB that corresponds to the second writer thread 104B of the source application 102. The array manager 108 identifies the second data array HOB corresponds to the second writer thread 104B among the data arrays H0A-N using the identifier of the second writer thread 104B. The array manager 108 identifies the second data array HOB corresponds to the second writer thread 104B based on the size of each data array and the location of each data array in the memory of the system 100. The system 100 inserts the received second input data tuple from the second writer thread 104B into the identified data array, i.e., the second data array HOB. The array manager 108 updates the size of the first data array H0A and the second data array HOB when the received first data tuple and second data tuple are inserted into the first data array H0A and the second data array HOB respectively. Preferably, the system 100 includes a read manager 112 and is communicatively connected to a consuming application 116 (e.g. a parallel data consumer, or an analytics system). The consuming application 116 includes a first reader thread 118A, a second reader thread 118B, and a Nthreader thread 118N. Each reader thread (e.g., the first reader thread 118A, the second reader thread 118B, and the Nlreader thread 118N) may include a numeric identifier, ranging from 0 to m- 1. The reader threads 118A-N are utilized to read the data tuples from the data arrays 110A-N.

[0049] The consuming application 116 initiates a read operation to read the data tuples from the data arrays 110A-N by requesting a read handle. The read handle is requested by each reader thread. The read handle determines a partition of the data tuples to be read by a specific reader thread using its thread identifier. The read handle includes an end index, and a current index that are specific to each data array of the data arrays 110A-N. For example, the first reader thread 118A of the consuming application 116 initializes the read handle using the read manager 112. The read handle initialized by the first reader thread 118A may be specifically related to the first data array 110A. This implies that the first reader thread 118A is assigned to the first data array 110A to read the data tuples in the first data array 110A. Similarly, the second reader thread 118B is assigned to the second data array HOB to read the data tuples in the second data array HOB. The read handle includes two indices, where the first index of the two indices corresponds to a position in a specific data array corresponding to the next data tuple to be read. Similarly, the second index of the two indices corresponds to a position in the specific data array which indicates that the next data array should be read from. The read handle is utilized to identify the assigned data array for each reader thread. Each reader thread then proceeds to access or read the data tuples within the assigned data array. The read manager 112 manages a read request from the consuming application 116. Upon identifying the assigned data array (e.g., the first data array HOA, or the second data array HOB) for each reader thread, a chunk manager 114 of the system 100 computes a collision-free chunking mechanism by dividing the data tuples that are stored in the first data array HOA into chunks such that each chunk is processed by each reader thread. The chunk manager 114 assigns each chunk to each reader thread, thereby avoiding collisions where the reader threads H8A-N access or read the same data tuple simultaneously in the array manager 108. The chunk manager 114 manages the partitioning of the data tuples in the data arrays H0A-N when reading is initiated by the consuming application 116

[0050] After identifying the assigned data array (e.g., the first data array HOA, or the second data array HOB) and chunk for the first reader thread H8A, the read manager 112 checks if there are any remaining data tuples in the chunk assigned to the first reader thread H8A. If the data tuples exist in the chunk, the first reader thread H8A reads the next data tuple from a position or an index (e.g., the first index or the second index) indicated by the read handle within the first data array HOA. If the data tuples are not in the chunk, the first reader thread H8A checks if the data tuples exist in the data arrays H0A-N. If the data tuples exist in the data arrays H0A-N, the chunk manager 114 performs the collision-free chunking mechanism to move to the next data array i.e., the second data array HOB using its identifier. Subsequently, the second reader thread H8B reads the data tuple from the second data array HOB.

[0051] FIG. 2 illustrates an operational flow of a system for ingesting data from a source application 202 in accordance with an implementation of the disclosure. The system includes a write manager 204, and an array manager 206. Preferably, the system is communicatively connected to the source application 202. At steps 208 and 210, the source application 202 processes a task of generating data tuples. The source application 202 stores the data tuples in one or more writer threads by assigning a writer thread identifier for the one or more writer threads. The write manager 204 receives the data tuples from the one or more writer threads of the source application 202. At a step 212, the write manager 204 identifies the corresponding data array among one or more data arrays in the array manager 206 for a received data tuple. The write manager 204 identifies the corresponding data array using the writer thread identifier of each data tuple. At a step 214, the array manager 206 inserts each data tuple into the corresponding data array. At a step 216, the source application 202 checks whether there is a task to generate data tuples. If the task is available to generate the data tuples, the source application 202 generates the data tuples. At a step 218, if no task exists, the source application 202 ends the process. FIG. 3 illustrates an operational flow of a system for reading data by a consuming application 302 in accordance with an implementation of the disclosure. The system includes a read manager 304, a chunk manager 306, and an array manager 308. Preferably, the system is communicatively connected to the consuming application 302. At a step 310, the consuming application 302 initiates a read operation. At a step 312, the read manager 304 requests to read the data tuples from data arrays using a read handle. At a step 314, the chunk manager 306 computes collision-free chunking mechanism using a reader thread. The chunk manager 306 divides the data tuples in the data arrays into a chunk format.

[0052] The chunk manager 306 computes a chunk size or a chunk format for the data tuples in each data array using, chunki= floor ((size of Di+m- 1) / m) where m represents counts of the one or more reader threads and Di represents the i11' data array.

[0053] Current = reader id *chunki where current is a current index or a position that indicates where the reader thread may start reading within the data array, Di, and chunki is the chunk size for the ithdata array Di.

[0054] The endi is calculated by, endi =min ((reader id+1)* chunki, size of Di)

[0055] The chunk size is calculated for each reader thread using the identifier of the reader. Where endi is an end index or a position that indicates where the reader thread should stop reading within the data array, Di For example, if m=2, the consuming application 302 includes two reader threads i.e., a first reader thread and a second reader thread.

[0056] At a step 316, the consuming application 302 requests the data tuples using the first reader thread and its corresponding read handle to read. At a step 318, the read manager 304 identifies the data array and chunk that the read handle should read. For example, the read manager 304 identifies a first data array and its corresponding chunk, which is assigned to the read handle. At a step of 320, the read manager 304 checks whether the data tuples in the first data array are in chunk format or not. At a step 322, the read manager 304 reads the data tuples from the first data array using the first reader thread if the data tuples are available in the chunk format. The read manager 304 reads the data tuples from the first data array based on the read handle of the first reader thread. At a step 324, the read manager 304 checks whether the data tuples are there in the array manager 308 or not if the data tuples in the first data array are not in the chunk format.

[0057] At a step 326, the chunk manager 306 updates or divides the data tuples in the next data array (i.e., a second data array) into the chunk format if the data tuples are there in the array manager 308 to read. The array manager 308 reads the data tuples from the second data array for the second reader thread identifier of the read handle if the data tuples are there in the chunk format. At a step 328, the read manager 304 updates the read handle when the first reader thread reads the data tuples from the first data array. The read manager 304 checks whether any data tuples in the first data array have not been read yet by the read handle. The array manager 308 allows the read handle to read the remaining data tuples in the first data array if any data tuples in the first data array have not been read yet by the read handle. At a step of 330, the consuming application 302 ends the reading of the data tuples if the data tuples are not there in the array manager 308.

[0058] Performance Evaluation Procedure: struct data_structure; for (int i=0; i< epoch: ++i) { int *sample_counts = generate_sample_counts(); struct sample **samples = generate_samples(); unsigned **delays = generate_delays ();

[0059] #pragma omp parallel

[0060] { int tid = omp_get_thread_num(); for (int j=0; j < sample_counts [tid]; ++j) { data_structure. write (samples [tid ][j]); busy sleep (delays [tid ] [j]);

[0061] }

[0062] }

[0063] #pragma omp parallel data_structure.read ()

[0064] }

[0065] FIGS. 4A-4C illustrate exemplary diagrams of different stages of data arrays during a write operation in accordance with an implementation of the disclosure. FIG. 4A depicts the “current state” of the data arrays. In the current state, a first data array, Do, includes a memory of size 0, a second data array, Di, includes a memory of size 0, and a third data array2, D2, includes a memory of size 0. If the first data array, Do, the second data array, Di, and the third data array2, D2, include the memory of size 0, it means the first data array, Do, the second data array, Di, and the third data array2, D2, do not include any data.

[0066] In FIG. 4B, the second data array, Di, is marked as bold, which means a second writer thread Ti utilizes its identifierto retrieve or identify the corresponding second data array, D 1. In FIG. 4C, the second data array, Di, is marked with black color, which means the second writer thread Ti writes or inserts the data tuple in the identified second data array, Di. Further, In FIG. 4C, a size of the second data array, Di, is marked as 1, which means the size of the second data array, Di, is updated after inserting the data tuple in the second data array, Di.

[0067] FIGS. 5 A and 5B illustrate exemplary diagrams of a read operation initiated by a consuming application in accordance with an implementation of the disclosure. In FIG. 5 A, a read handle is initialized to read a data tuple from a first data array, Do. The read handle includes a first reader thread and its two indexes (i.e., a current index and an end index). The first reader thread reads the data tuple using the read handle. The data tuple is read from the first data array, Do, using the current index of the first reader thread. For example, if the end index is 6 and the current index is 3 within the first data array, Do, an array manager (as shown in FIG. 1) checks whether the current index of the first reader thread is equal to the end index. If the current index of the first reader thread is not equal to the end index, the data tuple may be retrieved from the first data array, Do, corresponding to the third position in the first data array, Do, as shown in FIG. 5 A. Subsequently, the consuming application increases the current index by one. If the end index is 6 and the current index is 4 within the first data array, Do, the array manager checks whether the current index of the first reader thread is equal to the end index. If the current index of the first reader thread is not equal to the end index, the data tuple may be retrieved from the first data array, Do, corresponding to the fourth position in the first data array, Do, as shown in FIG. 5A.

[0068] The consuming application then again increments the current index by one. For example, if the end index is 6 and the current index is 5 within the first data array, Do, the array manager checks whether the current index of the first reader thread is equal to the end index. If the current index of the first reader thread is not equal to the end index, the data tuple may be retrieved from the first data array, Do, corresponding to the fifth position in the first data array, Do as shown in FIG. 5A. The consuming application then again increments the current index by one. If the current index of the first reader thread is equal to the end index, it indicates that the first reader thread has reached the end of the chunk in the first data array, Do. Consequently, the read handle starts reading the data tuple from a second data array, Di, using a second reader thread.

[0069] In FIG. 5B, the second reader thread initializes the read handle to read a data tuple from the second data array, Di. The read handle includes the second reader thread and its two indexes (i.e., a current index and an end index). The second reader thread reads the data tuple using the read handle. The data tuple is read from the second data array, Di, using the current index of the second reader thread.

[0070] For example, if the end index is 3 and the current index is 2 within the second data array, Di, the array manager checks whether the current index of the second reader thread is equal to the end index. If the current index of the second reader thread is not equal to the end index, the data tuple retrieved from the second data array, Di, corresponds to the second position in the second data array, Di, as shown in FIG. 5B. Subsequently, the consuming application increases the current index by one. If the end index is 3 and the current index is 3 within the second data array, Di, the array manager checks whether the current index of the first reader thread is equal to the end index. If the current index of the second reader thread is equal to the end index, it indicates that the second reader thread has reached the end of the chunk in the second data array, Di.

[0071] FIG. 6 is a flow diagram that illustrates a method of ingesting data from a source application in accordance with an implementation of the disclosure. At a step 602, the method includes receiving input data tuples from one or more writer threads of the source application. At a step 604, the method includes identifying a first data array amongst one or more data arrays for a received first input data tuple from a first writer thread of the one or more writer threads. The identified first data array corresponds to the first writer thread of the source application. At a step 606, the method includes inserting the received first input data tuple from the first writer thread into the identified first data array.

[0072] At a step 608, the method includes identifying a second data array amongst the one or more data arrays for a received second input data tuple from a second writer thread of the one or more writer threads. The identified second data arrays correspond to the second writer thread of the source application. At a step 610, the method includes inserting the received second input data tuple from the second writer thread into the identified second data array.

[0073] FIG. 7 is an illustration of a computer system (e.g., the system of FIG. 1) in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 700 includes at least one processor 704 that is connected to a bus 702, wherein the computer system 700 may be implemented using any suitable protocol, such as Peripheral Component Interconnect, PCI-Express, Accelerated Graphics Port, AGP, Hyper Transport, or any other bus or point-to-point communication protocol. The computer system 700 also includes a memory 706.

[0074] Control logic (software) and data are stored in the memory 706 which may take a form of random-access memory, RAM. In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit, CPU and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.

[0075] The computer system 700 may also include a secondary storage 710. The secondary storage 710 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk, a DVD drive, a recording device, a universal serial bus, a USB flash memory. The removable storage drives at least one of reads from and writes to a removable storage unit in a well-known manner. Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 706 and the secondary storage 710. Such computer programs, when executed, enable the computer system 700 to perform various functions as described in the foregoing. The memory 706, the secondary storage 710, and any other storage are possible examples of computer-readable media.

[0076] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 704, a graphics processor coupled to a communication interface 712, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 704 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0077] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated to entertainment purposes, an application-specific system. For example, the computer system 700 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, or an embedded system.

[0078] Furthermore, the computer system 700 may take the form of various other devices including, but not limited to a personal digital assistant, PDA, device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 700 may be coupled to a network (for example, a telecommunications network, a local area network, LAN, a wireless network, a wide area network, WAN, such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I / O interface 708.

[0079] It should be understood that the arrangement of components illustrated in the figures described is exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.

[0080] In addition, while at least one of these components is implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

[0081] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. A method of ingesting data from a source application (102, 202), comprising steps of: receiving input data tuples from a plurality of writer threads (104A-N) of the source application (102, 202); for a received first input data tuple from a first writer thread (104A) of the plurality of writer threads (104A-N), identifying a first data array (110A) amongst a plurality of data arrays (110A-N), wherein the identified first data array (110A) corresponds to the first writer thread (104A) of the source application (102, 202); inserting the received first input data tuple from the first writer thread (104 A) into the identified first data array (110A); for a received second input data tuple from a second writer thread (104B) of the plurality of writer threads (104A- N), identifying a second data array (HOB) amongst the plurality of data arrays (110A-N), wherein the identified second data array (110B) corresponds to the second writer thread (104B) of the source application (102, 202); and inserting the received second input data tuple from the second writer thread (104B) into the identified second data array (HOB).

2. The method of claim 1, wherein a size of each of the first and second data arrays (110A and 110B) is updated after the received first and second input data tuples is inserted into the first and second data arrays (110A and 110B), respectively.

3. The method of claim 1 , wherein a writer thread identifier is used to identify the first and second writer threads ( 104 A and 104B).

4. The method of claim 3, wherein the writer thread identifier is a numeric identifier.

5. The method of claim 4, wherein each of the plurality of data arrays (110A-N) is identified using an identifier with a numeric value corresponding to a value of the writer thread identifier.

6. The method of claim 1, wherein a consuming application (116, 302) having a plurality of reader threads (118A-N) reads data from the plurality of data arrays (110A-N).

7. The method of claim 6, wherein the consuming application (116, 302) initiates a read operation from the plurality of data arrays (110A-N) by requesting a read handle.

8. The method of claim 7, wherein the read handle is initialised by a first reader thread (118A) of the consuming application (H6).

9. The method of claim 8, wherein the read handle initialised by the first reader thread (118A) contains a reference to a specific data array of the plurality of data arrays (110A-N).

10. The method of claim 9, wherein the read handle initialised by the first reader thread (118A) contains two indices, where a first index of the two indices corresponds to a position in the specific data array corresponding to a next data tuple to be read, and a second index of the two indices corresponds to a position in the specific data array which indicates that a next data array should be read from.

11. The method of claim 9, wherein the first reader thread (118A) of the consuming application (116, 302) reads a data tuple, wherein the data tuple that is read is identified by the data array and the first index of the read handle.

12. The method of claim 11, wherein, after reading the data tuple, the first reader thread (118A) of the consuming application (116, 302) updates the read handle by increasing the first index by a value of one or updates the read handle by modifying the first index so that it points to a next data tuple to read.

13. The method of claim 12, wherein, if the first index is equal to the second index, the read handle is updated with the next data array.

14. A system (100) comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

15. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.

Images