Methods and systems for ingesting arrays from source applications
By assigning identifiers to write threads and inserting data arrays independently, combined with a conflict-free block mechanism, the problem of concurrent access management for data ingestion in large-scale parallel applications is solved, improving performance and analysis efficiency.
Patent Information
- Application Number
- CN202480091092.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from concurrency management complexity and scalability bottlenecks when handling data ingestion in large-scale parallel applications. Especially in highly competitive scenarios, existing methods such as lock-based and lock-free algorithms experience performance degradation under multi-threaded access, and the complexity of data analysis increases.
By assigning a unique identifier to each write thread, inserting data tuples into the corresponding data array, and accessing them independently through read handles, the synchronization requirements between write and read threads are eliminated, and a conflict-free block mechanism is used to manage read operations.
It improves data ingestion and analysis performance, ensuring efficient data processing and management, and is suitable for large-scale traffic simulation data ingestion.
Smart Images

Figure CN123125586A_ABST
Abstract
Description
Technical Field
[0001] This invention generally relates to data processing in massively parallel applications, and more specifically, to methods and systems for ingesting arrays from a source application. Background Technology
[0002] Modern large-scale applications generate massive amounts of data that need to be processed to derive or extract meaningful insights. These modern large-scale applications typically cover diverse domains, including large-scale traffic simulations that model the movement of vehicles, pedestrians, and other elements within a road network. For example, large-scale traffic simulations continuously generate data over time, providing information and insights into various aspects of the simulated environment, such as the behavior of vehicles, traffic lights, road signs, and pedestrians. Despite the sheer volume of information, it often lacks the specificity required for actionable insights. For instance, simply knowing the spatial coordinates (x, y) of a vehicle at any given moment may not be sufficient to understand the overall traffic dynamics.
[0003] To extract actionable or meaningful insights from this data, it needs to be processed, which involves calculating metrics relevant to domain experts. These metrics could include average vehicle speeds on key roads, or the location and magnitude of traffic congestion. By calculating these metrics, experts can gain a comprehensive understanding of traffic dynamics, including vehicle movement, changes in traffic flow, congestion patterns, and overall behavior.
[0004] Similarly, other software systems (such as database systems, performance monitoring systems, and logging systems) can follow similar principles. These systems also generate massive amounts of data that require analysis and processing to extract meaningful / valuable insights before they can be considered fully operational or usable. However, the sheer volume and complexity of the data present significant challenges in performing such analysis.
[0005] Furthermore, software tools such as Excel or MATLAB are unsuitable for processing and handling massive amounts of data due to their limitations in scale and performance. Traditional post-processing paradigms involve collecting massive amounts of data on disk and then performing analytical processing tasks on disk to gain valuable insights. However, this approach cannot perform real-time analysis and processing of data while the software system is generating it.
[0006] Stream processing has emerged as a viable alternative, leveraging specialized tools like data stream processing engines. These engines, comprising ingestion, processing, and output systems, simplify the processing pipeline. Nevertheless, the flow of data between pipeline stages can introduce complexity, particularly when multiple actors or threads are involved. Furthermore, maintaining guarantees such as preserving data order presents computational challenges, especially in high-concurrency scenarios.
[0007] In contrast, some massively parallel applications process massive amounts of data in different stages, employing a staged processing approach that divides tasks into different phases for efficient processing. This approach allows multiple threads to execute tasks concurrently, generating output data consisting of logically concurrent tuples. Therefore, the order in which these tuples are processed does not need to be strictly ordered in the ingestion system. In other words, the tuples do not need to be processed strictly according to the order in which they are received.
[0008] For example, when updating the simulation data of vehicles from simulation time (t0) to the next time (t1), the state of each simulated vehicle needs to be updated accordingly. The simulation data can include the position and speed of each vehicle. In multi-threaded update computations, the order in which a particular vehicle (A) is updated relative to another vehicle (B) is irrelevant to analyzing the simulation behavior at a given simulation time. In this case, applying strict ordering and synchronization can degrade the performance of massively parallel applications, leading to the complexity of managing massive amounts of data and complicating the analysis of simulation behavior at specific points in time.
[0009] Despite progress, managing concurrent data access remains a critical issue. Existing systems employ either locking schemes or lock-free algorithms to handle concurrent reads and writes of tuples. Locking schemes use locks to manage access to internal data structures, meaning only one thread can access the data at a time. Lock-free algorithms manage concurrency at a finer granular level without relying on locks. While both approaches support concurrent access, they both face scalability bottlenecks in high-contention scenarios. In high-contention scenarios, many threads concurrently access data over extended periods. For example, locking schemes may encounter scalability bottlenecks with more than 2 to 3 concurrent threads, while lock-free methods may face scalability bottlenecks with approximately 8 to 10 threads.
[0010] Existing technologies such as Nebula Stream and Apache Kafka offer alternative approaches to managing data concurrency. Nebula Stream leverages non-blocking queues for multi-threaded access, while Apache Kafka partitions data to facilitate parallel processing. However, these methods have limitations because they impose a trade-off on users and may not fully address the parallelism challenges of individual ingestion systems. Furthermore, they introduce complexity, particularly in scenarios requiring logical grouping of data. For example, when tracking website purchases, partitioning orders based on price can make it difficult to distinguish orders containing similar products but different quantities. This is because partitioning is autonomous and can complicate data analysis.
[0011] Therefore, it is necessary to address the aforementioned technical issues / deficiencies in processing data from the source application. Summary of the Invention
[0012] One object of the present invention is to provide a method and system for ingesting arrays from a source application while avoiding one or more disadvantages of existing methods.
[0013] The above objectives are achieved through the features of the independent claims. Further implementations are apparent in the dependent claims, the specification, and the drawings.
[0014] According to a first aspect, a method is provided for ingesting an array from a source application. The method includes receiving an input data tuple from one or more write threads of the source application. The method includes: identifying a first data array among one or more data arrays from a first write thread of the one or more write threads. The identified first data array corresponds to the first write thread of the source application. The method includes inserting the first input data tuple received from the first write thread into the identified first data array. The method includes identifying a second data array among the one or more data arrays from a second input data tuple received from a second write thread of the one or more write threads. The identified second data array corresponds to the second write thread of the source application. The method includes inserting the second input data tuple received from the second write thread into the identified second data array.
[0015] The method eliminates the need for synchronization between one or more write threads during data ingestion. This elimination enhances the performance of high-performance applications by removing the necessity for synchronization between the one or more write threads, thereby improving ingestion performance. Large-scale traffic simulation data ingestion based on the CityMoS application effectively meets the needs of stakeholders and ensures efficient data handling and processing.
[0016] Preferably, after the received first input data tuple and second input data tuple are respectively inserted into the first data array and the second data array, the size of each data array in the first data array and the second data array is updated.
[0017] Preferably, the write thread identifier is used to identify the first write thread and the second write thread. Preferably, the write thread identifier is a numeric identifier. Preferably, each of the plurality of data arrays is identified using an identifier having a value corresponding to the write thread identifier.
[0018] Preferably, a consumer application with multiple read threads reads data from the plurality of data arrays. Preferably, the consumer application initiates a read operation from the plurality of data arrays by requesting a read handle. Preferably, the read handle is initialized by a first read thread of the consumer application. Preferably, the read handle initialized by the first read thread contains a reference to a specific data array among the plurality of data arrays.
[0019] Preferably, the read handle initialized by the first read thread includes two indices, wherein the first index corresponds to the position in the specific data array corresponding to the next data tuple to be read, and the second index corresponds to the position in the specific data array indicating from which the next data array should be read. Preferably, the first read thread of the consumer application reads the data tuple. The read data tuple is identified by the data array and the first index of the read handle.
[0020] Preferably, after reading the data tuple, the first read thread of the consuming application updates the read handle by incrementing the first index by 1. Preferably, if the first index is equal to the second index, the read handle is updated to the next data array.
[0021] The method allows the one or more read threads to access the system independently during data consumption without centralized synchronization. Furthermore, the method eliminates the need for synchronization between the one or more read threads, thereby improving analytical computing performance.
[0022] According to the second aspect, a system includes modules for performing all steps of the provided method.
[0023] According to a third aspect, a computer program includes instructions for performing all the steps of the above-described method when the computer program is executed on a computer system.
[0024] These and other aspects of the invention will be apparent from the implementation described below. Attached Figure Description
[0025] The various implementations of the present invention will now be described by way of example and in conjunction with the accompanying drawings, wherein: Figure 1 A block diagram of a system for ingesting arrays from a source application, provided for an implementation of the present invention; Figure 2 The present invention illustrates the operational flow of a system for ingesting arrays from a source application, as provided by an implementation of the present invention. Figure 3 The present invention illustrates the operation flow of a system for reading data by a consumer application, provided by an implementation of the present invention; Figures 4A to 4C An exemplary diagram is shown illustrating different stages of a data array during a write operation provided by an implementation of the present invention; Figure 5A and Figure 5B An exemplary diagram is shown, illustrating a read operation initiated by a consumer application, provided by an implementation of the present invention. Figure 6 A flowchart illustrating a method for ingesting an array from a source application, as provided in this embodiment of the invention; Figure 7 A diagram of a computing system in which various architectures and functions of various previous implementation methods can be realized. Detailed Implementation
[0026] The present invention provides a method and system for ingesting arrays from a source application.
[0027] To enable those skilled in the art to more easily understand the present invention, the following implementation of the present invention is described in conjunction with the accompanying drawings.
[0028] The terms “first,” “second,” “third,” and “fourth” (if any) used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular sequence or order. It should be understood that such terms are interchangeable where appropriate, for example, to allow implementations of the invention described herein to be implemented in sequences other than those shown or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to the steps or units expressly listed, but may include other steps or units not expressly listed or inherent to such processes, methods, products, or apparatuses.
[0029] definition: Thread: A thread is a program consisting of a series of computer instructions designed to perform a specific task (e.g., changing a flat tire on a car). For example, the process may include (i) finding a safe place to park, (ii) using tools to lift the car, (iii) removing the punctured tire, (iv) installing and tightening a new tire, and (iv) lowering the car back to the ground.
[0030] Now, if two tires need to be changed, steps (iii) and (iv) are repeated sequentially. Alternatively, if other passengers are present, they can independently change the other punctured tire while the driver has completed the first tire change, performing steps (iii) and (iv) separately, thus reducing the time required to complete the repair. Similarly, in computer programs, threads can execute tasks in parallel. In other words, threads are essentially concurrent and independently running streams of computational operations, enabling efficient use of computing resources and faster task completion.
[0031] Tuples: A tuple is typically defined as a single row or record in a table that encapsulates a set of related attributes. Each tuple includes multiple fields or columns, each corresponding to a specific data element or characteristic. Conceptually, it can be thought of as horizontal blocks of data in a table, where each field holds a value representing a specific attribute or property. For example, in the context of a database table on an e-commerce website, a tuple can represent detailed information about a single product. It typically includes fields such as a unique numeric product identifier, product name, description, price, and availability status. Each tuple in the table contains distinct values corresponding to the attributes of each product, facilitating the organized storage and retrieval of product information.
[0032] Data arrays: A data array is a structured collection of data elements (such as values or variables) that are arranged in order and stored contiguously in the system's memory.
[0033] Figure 1A block diagram of a system 100 for ingesting an array from a source application 102, provided for implementation of the present invention. System 100 includes a write manager 106 and an array manager 108. The source application 102 (e.g., a parallel data generator) includes a first write thread 104A, a second write thread 104B, and an Nth write thread 104N. Preferably, system 100 is communicatively connected to the source application 102. The source application 102 may be a data producer. Each write thread (e.g., the first write thread 104A, the second write thread 104B, and the Nth write thread 104N) includes a data tuple of the source application 102 having a write thread identifier. The write thread identifier may be a numeric identifier between 0 and n–1. The source application 102 assigns the write thread identifier to each write thread to distinguish each write thread from other write threads. The write thread identifier can be used to identify the first write thread 104A and the second write thread 104B. This means that the write thread identifier distinguishes the initial write thread from subsequent write threads. Each write thread uses source application 102 to collect data tuples from one or more external sources. The data tuples may include a first data tuple, a second data tuple, and an nth data tuple. Source application 102 generates the data tuples by simulating events within source application 102 (e.g., traffic simulations, etc.). Array manager 108 includes a first data array 110A, a second data array 110B, and an nth data array 110N. Each data array (e.g., the first data array 110A, the second data array 110B, and the nth data array 110N) includes a numeric identifier ranging from 0 to n–1.
[0034] Write manager 106 receives a first data tuple from a first write thread 104A of source application 102 and a second data tuple from a second write thread 104B. Write manager 106 transfers the first data tuple received from the first write thread 104A to array manager 108. Array manager 108 identifies a first data array 110A among the data arrays 110A to 110N corresponding to the first write thread 104A of source application 102. Array manager 108 identifies the first data array 110A corresponding to the first write thread 104A among the data arrays 110A to 110N using the write thread identifier of the first write thread 104A. Preferably, array manager 108 identifies the first data array 110A corresponding to the first write thread 104A among the data arrays 110A to 110N based on the size of each data array 110A to 110N and the location of each data array in the memory of system 100. Each data array is identified using an identifier corresponding to each data array. The identifier may include a numerical value corresponding to a write thread identifier associated with the identifier. System 100 inserts a first input data tuple received from a first write thread 104A into an identified first data array 110A. During the insertion process, while write threads 104A to 104N are active, the mapping of data arrays 110A to 110N remains unaffected by the order of the data tuples. For example, if a third data array 2110C is identified for the first data tuple, then the first data tuple is inserted into the third data array 2110C.
[0035] Write manager 106 transfers a second data tuple from second write thread 104B to array manager 108. Array manager 108 identifies a second data array 110B corresponding to the second write thread 104B of source application 102. Array manager 108 uses the identifier of second write thread 104B to identify the second data array 110B corresponding to the second write thread 104B among data arrays 110A to 110N. Array manager 108 identifies the second data array 110B corresponding to the second write thread 104B based on the size of each data array and the location of each data array in the memory of system 100. System 100 inserts the second input data tuple received from second write thread 104B into the identified data array, i.e., the second data array 110B. When the received first data tuple and second data tuple are inserted into the first data array 110A and the second data array 110B, respectively, array manager 108 updates the size of the first data array 110A and the second data array 110B.
[0036] Preferably, system 100 includes a read manager 112 and is communicatively connected to a consumer application 116 (e.g., a parallel data consumer or analytics system). Consumer application 116 includes a first read thread 118A, a second read thread 118B, and an Nth read thread 118N. Each read thread (e.g., the first read thread 118A, the second read thread 118B, and the Nth read thread 118N) may include a numeric identifier ranging from 0 to m–1. Read threads 118A to 118N are used to read data tuples from data arrays 110A to 110N.
[0037] Consumer application 116 initiates a read operation by requesting a read handle to read data tuples from data arrays 110A to 110N. Each read thread requests a read handle. The read handle uses its thread identifier to determine the partition of the data tuples that a particular read thread will read. The read handle includes a specific end index and a current index for each data array in data arrays 110A to 110N. For example, the first read thread 118A of consumer application 116 initializes a read handle using read manager 112. The read handle initialized by the first read thread 118A may specifically be associated with the first data array 110A. This means that the first read thread 118A is assigned to the first data array 110A to read data tuples in the first data array 110A. Similarly, the second read thread 118B is assigned to the second data array 110B to read data tuples in the second data array 110B. The read handle includes two indices, the first of which corresponds to the position in the specific data array corresponding to the next data tuple to be read. Similarly, the second index of the two indices corresponds to a position in a specific data array that indicates from which the next data array should be read. A read handle is used to identify the data array allocated to each read thread. Each read thread then continues to access or read data tuples in its allocated data array. Read manager 112 manages read requests from consumer application 116. When identifying the data array allocated to each read thread (e.g., first data array 110A or second data array 110B), block manager 114 of system 100 computes a conflict-free chunking mechanism by dividing the data tuples stored in first data array 110A into chunks, such that each chunk is processed by each read thread. Block manager 114 assigns each chunk to each read thread, thereby avoiding conflicts where read threads 118A to 118N simultaneously access or read the same data tuples in array manager 108. When a read is initiated by consumer application 116, block manager 114 manages the partitioning of data tuples in data arrays 110A to 110N.
[0038] After identifying the data array (e.g., first data array 110A or second data array 110B) and block allocated to the first read thread 118A, the read manager 112 checks whether any remaining data tuples exist in the block allocated to the first read thread 118A. If the data tuple exists in the block, the first read thread 118A reads the next data tuple from the position or index (e.g., the first index or the second index) indicated by the read handle in the first data array 110A. If the data tuple is not in the block, the first read thread 118A checks whether the data tuple exists in data arrays 110A through 110N. If the data tuple exists in data arrays 110A through 110N, the block manager 114 performs a conflict-free block partitioning mechanism to move to the next data array, i.e., the second data array 110B, using its identifier. Subsequently, the second read thread 118B reads the data tuple from the second data array 110B.
[0039] Figure 2 The following diagram illustrates the operational flow of a system for ingesting an array from a source application 202, provided by an implementation of the present invention. The system includes a write manager 204 and an array manager 206. Preferably, the system is communicatively connected to the source application 202. In steps 208 and 210, the source application 202 processes the task of generating data tuples. The source application 202 stores data tuples in one or more write threads by assigning write thread identifiers to one or more write threads. The write manager 204 receives data tuples from one or more write threads of the source application 202. In step 212, the write manager 204 identifies the corresponding data array in one or more data arrays in the array manager 206 for each received data tuple. The write manager 204 uses the write thread identifier of each data tuple to identify the corresponding data array. In step 214, the array manager 206 inserts each data tuple into the corresponding data array. In step 216, the source application 202 checks whether a task for generating data tuples exists. If a task is available for generating data tuples, the source application 202 generates the data tuples. In step 218, if no task exists, the source application 202 terminates the process.
[0040] Figure 3 The following diagram illustrates the operational flow of a system for reading data by a consumer application 302, provided by an implementation of the present invention. The system includes a read manager 304, a block manager 306, and an array manager 308. Preferably, the system is communicatively connected to the consumer application 302. In step 310, the consumer application 302 initiates a read operation. In step 312, the read manager 304 requests to read data tuples from a data array using a read handle. In step 314, the block manager 306 uses a read thread to compute a conflict-free block partitioning mechanism. The block manager 306 partitions the data tuples in the data array into block formats.
[0041] Block manager 306 uses the following method to calculate the block size or block format of each data tuple in the data array: chunk i = floor ((size of D i +m– 1) / m) Where m represents the number of one or more read threads, and D i This represents the i-th data array.
[0042] Current i = reader id chunk i Where current i Is the current index or indicator that the read thread can access the data array D? i The starting position for reading from the chunk. i It is the i-th data array D i The block size.
[0043] end i The calculation formula is: end i =min ((reader id+1) chunk i size of D i ) The block size is calculated for each read thread using the reader's identifier. Where, end i Is it the end index or an indication that the read thread should be in the data array D? i The position where reading stops. For example, if m=2, then consumer application 302 includes two read threads, namely, the first read thread and the second read thread.
[0044] In step 316, consumer application 302 requests to read data tuples using a first read thread and its corresponding read handle. In step 318, read manager 304 identifies the data array and blocks that the read handle should read. For example, read manager 304 identifies a first data array and its corresponding blocks allocated to the read handle. In step 320, read manager 304 checks whether the data tuples in the first data array are in block format. In step 322, if the data tuples are available in block format, read manager 304 reads the data tuples from the first data array using the first read thread. Read manager 304 reads data tuples from the first data array according to the read handle of the first read thread. In step 324, if the data tuples in the first data array are not in block format, read manager 304 checks whether the data tuples exist in array manager 308.
[0045] In step 326, if there are data tuples to be read in array manager 308, block manager 306 updates or partitions the data tuples in the next data array (i.e., the second data array) into block format. If the data tuples exist in block format, array manager 308 reads the data tuples from the second data array of the second read thread identifier of the read handle. In step 328, while the first read thread is reading data tuples from the first data array, read manager 304 updates the read handle. Read manager 304 checks if any data tuples in the first data array have not yet been read by the read handle. If any data tuples in the first data array have not yet been read by the read handle, array manager 308 makes the read handle available to read the remaining data tuples in the first data array. In step 330, if the data tuples are not in array manager 308, consumer application 302 terminates the reading of the data tuples.
[0046] Performance evaluation procedure: struct data_structure; for (int i=0; i <epoch: ++i) { int sample_counts = generate_sample_counts(); struct sample samples = generate_samples(); unsigned delays = generate_delays(); #pragma omp parallel { int tid = omp_get_thread_num(); for (int j=0; j <sample_counts [tid]; ++j) { data_structure.write (samples [tid ][j]); busy sleep (delays [tid ] [j]); } } #pragma omp parallel data_structure.read() } Figures 4A to 4C An exemplary diagram is shown illustrating different stages of a data array during a write operation provided by an implementation of the present invention. Figure 4A This describes the "current state" of the data arrays. In the current state, the first data array D0 includes memory of size 0, the second data array D1 includes memory of size 0, and the third data array 2D2 includes memory of size 0. If the first data array D0, the second data array D1, and the third data array 2D2 include memory of size 0, it means that the first data array D0, the second data array D1, and the third data array 2D2 do not contain any data.
[0047] exist Figure 4B In the diagram, the second data array D1 is marked in bold, meaning that the second write thread T1 uses its identifier to retrieve or identify the corresponding second data array D1. Figure 4C In the diagram, the second data array D1 is marked in black, indicating that the second write thread T1 will write or insert data tuples into or into the identified second data array D1. Furthermore, in... Figure 4C In the diagram, the size of the second data array D1 is marked as 1, which means that the size of the second data array D1 is updated after a data tuple is inserted into it.
[0048] Figure 5A and Figure 5B An exemplary diagram is shown illustrating a read operation initiated by a consumer application, provided by an implementation of the present invention. Figure 5AIn this context, the read handle is initialized to read data tuples from the first data array D0. The read handle includes the first read thread and its two indices (i.e., the current index and the end index). The first read thread uses the read handle to read the data tuples. The data tuples are read from the first data array D0 using the current index of the first read thread. For example, if the end index in the first data array D0 is 6 and the current index is 3, then the array manager (e.g., ...) will read the data tuples from the first data array D0. Figure 1 (As shown) Check if the current index of the first read thread is equal to the end index. If the current index of the first read thread is not equal to the end index, a data tuple can be retrieved from the first data array D0 corresponding to the third position in the first data array D0, such as... Figure 5A As shown. Subsequently, the consumer application increments the current index by one. If the end index in the first data array D0 is 6 and the current index is 4, the array manager checks if the current index of the first read thread is equal to the end index. If the current index of the first read thread is not equal to the end index, a data tuple can be retrieved from the first data array D0 corresponding to the fourth position in the first data array D0, such as... Figure 5A As shown.
[0049] Then, the consuming application increments the current index again. For example, if the end index in the first data array D0 is 6 and the current index is 5, the array manager checks if the current index of the first read thread is equal to the end index. If the current index of the first read thread is not equal to the end index, a data tuple can be retrieved from the first data array D0 corresponding to the fifth position in the first data array D0, such as... Figure 5A As shown. Then, the consuming application increments the current index again. If the current index of the first read thread equals the end index, it means that the first read thread has reached the end of the block in the first data array D0. Therefore, the read handle begins to read data tuples from the second data array D1 using the second read thread.
[0050] exist Figure 5B In this process, the second read thread initializes a read handle to read data tuples from the second data array D1. The read handle includes the second read thread and its two indices (i.e., the current index and the end index). The second read thread uses the read handle to read the data tuples. The second read thread's current index is used to read data tuples from the second data array D1.
[0051] For example, if the end index in the second data array D1 is 3 and the current index is 2, the array manager checks whether the current index of the second read thread is equal to the end index. If the current index of the second read thread is not equal to the end index, the data tuple retrieved from the second data array D1 corresponds to the second position in the second data array D1, such as... Figure 5BAs shown in the diagram. The consuming application then increments the current index by one. If the end index in the second data array D1 is 3 and the current index is 3, the array manager checks if the current index of the first read thread is equal to the end index. If the current index of the second read thread is equal to the end index, it indicates that the second read thread has reached the end of the block in the second data array D1.
[0052] Figure 6 A flowchart illustrating a method for ingesting an array from a source application, provided for an implementation of the present invention. In step 602, the method includes receiving an input data tuple from one or more write threads of the source application. In step 604, the method includes identifying a first data array within one or more data arrays of the first input data tuple received by a first write thread in one of the one or more write threads. The identified first data array corresponds to the first write thread of the source application. In step 606, the method includes inserting the first input data tuple received from the first write thread into the identified first data array.
[0053] In step 608, the method includes identifying a second data array in the one or more data arrays for a second input data tuple received from a second write thread of one or more write threads. The identified second data array corresponds to the second write thread of the source application. In step 610, the method includes inserting the second input data tuple received from the second write thread into the identified second data array.
[0054] Figure 7 For computer systems with various architectures and functions that can implement various previous implementation methods (e.g., Figure 1 The diagram illustrates a computer system 700. As shown, the computer system 700 includes at least one processor 704 connected to a bus 702. The computer system 700 can be implemented using any suitable protocol, such as the Peripheral Component Interconnect (PCI) standard, PCI-Express, Accelerated Graphics Port (AGP), HyperTransport, or any other bus or point-to-point communication protocol. The computer system 700 also includes a memory 706.
[0055] The control logic (software) and data are stored in memory 706, which may take the form of random-access memory (RAM). In this invention, a single semiconductor platform can refer to a unique integrated circuit or chip based on a single semiconductor. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with increased connectivity, simulating an on-chip module with increased connectivity, simulating on-chip operations, and representing a substantial improvement over traditional implementations using a central processing unit (CPU) and bus. Of course, various modules can also be placed individually or in various combinations of semiconductor platforms, depending on the user's preference.
[0056] The computer system 700 may also include auxiliary storage 710. For example, auxiliary storage 710 includes hard disk drives and removable storage drives, representing floppy disk drives, magnetic tape drives, compact disk drives, digital versatile disk (DVD) drives, recording devices, and universal serial bus (USB) flash memory. The removable storage drive reads and writes at least one of the removable storage units in a well-known manner.
[0057] Computer programs or computer control logic algorithms may be stored in memory 706 and / or auxiliary memory 710. When these computer programs are executed, they enable computer system 700 to perform the various functions described above. Memory 706, auxiliary memory 710, and any other memory are possible examples of computer-readable media.
[0058] In one implementation, the architecture and functionality described in the various previous figures may be in processor 704, graphics processor coupled to communication interface 712, integrated circuit (not shown) capable of having at least a portion of the capabilities of both processor 704 and graphics processor, and chipset (i.e., a set of integrated circuits designed to operate and be marketed as units performing related functions, etc.).
[0059] Furthermore, the architectures and functions described in the various preceding figures can be implemented within the context of general-purpose computer systems, circuit board systems, game console systems dedicated to entertainment purposes, and special-purpose systems. For example, computer system 700 can take the form of a desktop computer, laptop computer, server, workstation, game console, or embedded system.
[0060] Furthermore, the computer system 700 can take the form of various other devices, including but not limited to personal digital assistant (PDA) devices, mobile phone devices, smartphones, televisions, etc. Additionally, although not shown, the computer system 700 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN), such as the Internet, a peer-to-peer network, a cable network, etc.) via I / O interface 708 for communication purposes.
[0061] It should be understood that the arrangement of components shown in the described figures is exemplary and other arrangements are possible. It should also be understood that the various system components (and modules) defined by the following claims and shown in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and modules) may be implemented wholly or partially by at least some of the parts shown in the arrangements shown in the described figures.
[0062] Furthermore, while at least one of these components is implemented at least partially as an electronic hardware component and thus constitutes a machine, the other components may be implemented in software, which, when included in the execution environment, constitutes a machine, hardware, or a combination of software and hardware.
[0063] Although the invention and its advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims.
Claims
1. A method for ingesting an array from a source application (102, 202), characterized in that, Includes the following steps: Input data tuples are received from multiple write threads (104A to 104N) of the source application (102, 202); For a first input data tuple received from a first write thread (104A) among the plurality of write threads (104A to 104N), a first data array (110A) is identified from a plurality of data arrays (110A to 110N), wherein the identified first data array (110A) corresponds to the first write thread (104A) of the source application (102, 202). The first input data tuple received from the first write thread (104A) is inserted into the first data array (110A) of the identifier; For a second input data tuple received from a second write thread (104B) among the plurality of write threads (104A to 104N), identify a second data array (110B) among the plurality of data arrays (110A to 110N), wherein the identified second data array (110B) corresponds to the second write thread (104B) of the source application (102, 202). The second input data tuple received from the second write thread (104B) is inserted into the second data array (110B) of the identifier.
2. The method according to claim 1, characterized in that, After the received first input data tuple and second input data tuple are respectively inserted into the first data array (110A) and the second data array (110B), the size of each data array in the first data array (110A) and the second data array (110B) is updated.
3. The method according to claim 1, characterized in that, The write thread identifier is used to identify the first write thread (104A) and the second write thread (104B).
4. The method according to claim 3, characterized in that, The write thread identifier is a numeric identifier.
5. The method according to claim 4, characterized in that, Each of the plurality of data arrays (110A to 110N) is identified using an identifier having a value corresponding to the write thread identifier.
6. The method according to claim 1, characterized in that, Consumer applications (116, 302) with multiple read threads (118A to 118N) read data from the multiple data arrays (110A to 110N).
7. The method according to claim 6, characterized in that, The consumer applications (116, 302) initiate a read operation from the plurality of data arrays (110A to 110N) by requesting a read handle.
8. The method according to claim 7, characterized in that, The read handle is initialized by the first read thread (118A) of the consumer application (116).
9. The method according to claim 8, characterized in that, The read handle, initialized by the first read thread (118A), contains a reference to a specific data array among the plurality of data arrays (110A to 110N).
10. The method according to claim 9, characterized in that, The read handle initialized by the first read thread (118A) contains two indices, wherein the first index of the two indices corresponds to the position in the specific data array corresponding to the next data tuple to be read, and the second index of the two indices corresponds to the position in the specific data array indicating from which the next data array should be read.
11. The method according to claim 9, characterized in that, The first read thread (118A) of the consumer application (116, 302) reads a data tuple, wherein the read data tuple is identified by the data array and the first index of the read handle.
12. The method according to claim 11, characterized in that, After reading the data tuple, the first read thread (118A) of the consumer application (116, 302) updates the read handle by incrementing the first index by 1, or by modifying the first index to point to the next data tuple to be read.
13. The method according to claim 12, characterized in that, If the first index is equal to the second index, then the read handle is updated to the next data array.
14. A system (100), characterized in that, Includes modules suitable for performing all steps of the method according to any one of the preceding claims.
15. A computer program, characterized in that, The instructions include, when the computer program is executed in a computer system, instructions for performing all steps of the method according to any one of the preceding claims.
