Data operations for commands in interactive programming session

Through the FAM dataset storage manager, the memory operations of multiple computer nodes are managed in an interactive programming session, and the problems of large dataset processing and resource allocation are solved, and efficient data sharing and parallel processing are realized.

CN119065802APending Publication Date: 2024-12-03HEWLETT PACKARD ENTERPRISE DEV LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410431987.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-10-26
Filing Date
2024-04-11
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In interactive programming sessions, programmers face challenges in local memory allocation and management across multiple computer nodes when processing large data sets, making it difficult to predict data volume and operational characteristics, resulting in difficulty in resource allocation and data sharing.

Method used

Introduced the FAM dataset storage manager to manage program execution during interactive programming sessions, as a server of remote computers and a client of fiber-attached memory, responsible for data replication, incremental updates and sharing, through parallel computing servers and multiple computers Distributed memory operation of nodes, supporting interactive programming and data sharing of non-interactive programs.

Benefits of technology

Improves the parallelism and processing efficiency of interactive programming, allowing programmers to perform high-performance computing operations on large data sets, effectively utilize computing resources, and support cross-node storage and sharing of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065802A_ABST
    Figure CN119065802A_ABST
Patent Text Reader

Abstract

The invention relates to data operations for commands in an interactive programming session. In some examples, a system having a plurality of computer nodes receives a command based on program code of a program being developed in an interactive programming session. The system distributes data items from a network attached memory to distributed data objects of data in a node memory having a plurality of computer nodes. The dataset manager performs operations specified by the command on the distributed data object, the operations being performed in parallel on the plurality of computer nodes. The dataset manager generates derived data generated by operations on the distributed data object, the derived data being accessible by a programmer in the interactive programming session.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This invention was made with government support under Contract No. H98230-15-D-0022 / 0003 awarded by the Maryland Procurement Office. The government has certain rights in this invention.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 505,483, entitled “Data Management for Fabric-Attached Memory,” filed on June 1, 2023, under 35 U.S.C. §119(e), which is incorporated herein by reference. Background Art

[0004] A program comprising machine-readable instructions can be executed in a computer system. The program can be written in any of a variety of different programming languages. The source code of the program can be compiled into machine executable code by a compiler for execution. Alternatively, the source code of the program can be executed by an interpreter without first performing compilation. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Some embodiments of the present disclosure are described with reference to the following drawings.

[0006] Figure 1 is a block diagram of an arrangement including a client computer, a parallel computing server executing on a computer node, and a fabric attached memory (FAM), according to some examples.

[0007] Figure 2 , Figure 3 and Figure 4 Depicted are operations applied to a FAM dataset according to some examples.

[0008] Figure 5 is a block diagram of an arrangement including a client computer, a plurality of parallel computation servers executing on a subset of computer nodes, and a FAM according to a further example.

[0009] Figure 6 is a block diagram of modules of a parallel computing server according to some examples.

[0010] Figure 7 is a block diagram of an arrangement including an incremental data updater according to some examples.

[0011] Figure 8 is a flow diagram of a process according to some examples.

[0012] Fig. 9 is a block diagram of a system according to some examples.

[0013] Fig.10 is a block diagram of a storage medium storing machine-readable instructions according to some examples.

[0014] In all drawings, the same reference numerals denote similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the dimensions of some components may be exaggerated to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0015] Some programming languages ​​support an interactive mode in which lines of code of a program can be executed as the programmer adds them. Python is an example of such a programming language. When a programmer creates lines of code, the lines of code can be executed individually or in batches during an interactive programming session. During an interactive programming session, a line of code can be executed against data stored in memory. As the amount of data to which operations are to be applied increases (such as operations associated with data mining, data analysis, artificial intelligence, or other operations), programmers may face challenges associated with timely execution of operations on large amounts of data. In some cases, clusters of computer nodes may be provided to increase processing power. However, the data to which any given operation is to be applied may not fit into the local memory of a single computer node. As a result, programmers may face challenges associated with how to allocate and manage data across the local memory of multiple computer nodes. In addition, such as sharing the resulting data resulting from the operations applied to the underlying data with other programmers may be challenging. In addition, deciding how many computer nodes to allocate to a programmer for an interactive programming session may be challenging because it is difficult to predict what operations will be performed in an interactive programming session and the amount of data that any operation will process or generate. In addition, different operations performed by programmers during an interactive session may have different characteristics and therefore have different requirements for effectively processing them.

[0016] According to some embodiments of the present disclosure, a system including multiple computer nodes executes a manager having the following roles: (1) the manager acts as a server for programs being written during an interactive programming session (such as in a remote computer away from the system), and (2) the manager acts as a client with respect to a network attached storage (also known as "fiber attached storage" or FAM). Such a manager is referred to below as a "FAM dataset storage manager". Commands of lines of code are issued from a remote computer where the program is developed to the manager, which manages the parallel execution of the operations of the commands across multiple computer nodes. In addition, in its role as a client with respect to the FAM, the manager is able to interact with the FAM to copy data items from the FAM to a distributed data object, which includes base data distributed across the storage of the computer nodes. Derived data generated by operations applied to the base data of the distributed data object is stored by the manager to the FAM. The derived data is automatically updated incrementally as additional base data is received. In addition, the manager supports sharing of base data and derived data with other entities, such as non-interactively executed programs (e.g., scheduled jobs) or other programmers in other interactive programming sessions.

[0017] Furthermore, multiple managers can contribute to a single operation by executing on different parts of the same data set. These managers can operate concurrently, thereby increasing the parallelism of the executed operations. Furthermore, depending on the characteristics of the operations to be performed, more (or fewer) managers can contribute to the execution of a given operation.

[0018] Using the technology or mechanism according to some examples of the present disclosure, it is possible to achieve improvements in interactive programming techniques when processing large data sets. As part of program development, lines of code applied to large data sets stored in the FAM can be executed in an interactive programming session. Data from large data sets can be retrieved from the FAM by a manager into the local memory of a computer node, and operations corresponding to lines of code are performed on data distributed across the local memory of the computer node. These operations can generate derived data that can be accessed by the programmer of the program and can be shared with other entities (such as other programmers).

[0019] Figure 1 1 is a block diagram of an example arrangement 100 that includes a client computer 102, a collection of server computer nodes 120-1 through 120-N (N≥2) that perform operations of one or more workloads initiated from the client computer 102, and a FAM 106 that stores data processed by the workloads. The client computer 102 may be physically coupled to the computer nodes 120-1 through 120-N via a network, which may include a local area network (LAN), a wide area network (WAN), a public network such as the Internet, or any other type of communication link.

[0020] In other examples, other arrangements of components may be employed other than arrangement 100. Furthermore, while some examples discussed in this disclosure involve the use of Python, Arkouda, and Chapel, it should be noted that in other examples, other programming languages ​​and other server programs that support parallel execution of operations across computer nodes during an interactive programming session may be employed.

[0021] FAM 106 is implemented using a collection of memory devices including non-volatile memory devices or volatile memory devices or both. Examples of memory devices include any one or some combination of the following devices: dynamic random access memory (DRAM) devices, static random access memory (SRAM) devices, flash memory devices, or any other type of memory device.

[0022] In a further example, the FAM 106 may include a collection of memory nodes, where a memory node may include a memory server and one or more memory devices. A memory server in a memory node manages access to data in one or more memory devices of the memory node.

[0023] FAM 106 can be accessed by processes executing in computer nodes 120-1 to 120-N using remote direct memory access (RDMA). Computer nodes 120-1 to 120-N are coupled to FAM 106 via interconnect 124. RDMA allows a first computer, such as one of computer nodes 120-1 to 120-N, to access the memory of a second computer, such as a computer of FAM 106, without involving an operating system (OS) or a main processor of the second computer. Examples of interconnect 124 can include any of the following: a fast compute link (CXL) interconnect, a slingshot interconnect, an InfiniBand interconnect, or any high throughput, low latency interconnect.

[0024] Programmer 108 may develop program 110 at client computer 102. In some examples, program 110 is based on the Python programming language. In other examples, program 110 may be based on a different programming language that supports an interactive mode in which lines of code 114 of program 110 are executed as program 110 is written (developed) by programmer 108.

[0025] although Figure 1 Only one client computer 102 is shown, but in other examples there may be multiple client computers, which may be associated with different programmers.

[0026] A "client computer" may refer to any one or some combination of a desktop computer, a laptop computer, a tablet computer, a smart phone, or any other electronic device that supports interactive use by a user. A "server computer node" (or more simply, a "computer node") may refer to a desktop computer, a laptop computer, a tablet computer, a smart phone, a server computer, a communication node (e.g., a switch, a router, a gateway, or another type of communication node), or any other type of electronic device.

[0027] A "line of code" of a program may refer to any portion of a program that can be executed separately. For example, a line of code may include one program statement or a collection of multiple program statements.

[0028] In some examples, a collection of computer nodes 120-1 to 120-N can execute a program that supports parallel execution of operations across multiple computer nodes. For example, the program may include a parallel computing server 104 that can be executed across multiple computer nodes. In some examples, the parallel computing server 104 is an Arkouda server based on Chapel. Chapel is an open source programming language that supports parallel computing. Programs written using Chapel can run in a distributed computer system including multiple computer nodes (such as computer nodes 120-1 to 120-N).

[0029] although Figure 1 Only one parallel computing server 104 is shown, but in other examples, multiple parallel computing servers (e.g., multiple Arkouda servers) may execute across multiple computer nodes. In such examples, different Arkouda servers may execute on corresponding different subsets of computer nodes. For example, a first Arkouda server may execute on a first subset of computer nodes, and a second Arkouda server may execute on a second subset of computer nodes. The first subset of computer nodes may include computer nodes that are not part of the second subset of computer nodes, and vice versa. The first and second subsets of computer nodes may share at least one computer node. Alternatively, the first and second subsets of computer nodes do not share any computer nodes. As used herein, a "subset" of computer nodes may refer to a single computer node or a plurality of computer nodes.

[0030] Some software packages allow users to interactively distribute parallel computations on distributed data (in the form of data arrays, such as pdarray). An example of such a software package is the Arkouda package written in Chapel. A data array is an example of a distributed data object that can be distributed across computer nodes.

[0031] The computational core of Arkouda is the Chapel interpreter, which accepts a set of predefined commands from the client and uses Chapel's built-in capabilities for multi-site and multi-threaded execution. Multi-site execution refers to execution across multiple processing sites, where "processing site" refers to any distinct computational partition, such as a computer node or a portion of a computer node including the resources of computer nodes 120-1 to 120-N. Figure 1 In the example of , a processing site can refer to a computer node, a portion of a computer node (e.g., a group of processors or a group of processor cores), or a collection of computer nodes. As used herein, a "collection" of items can refer to a single item or multiple items.

[0032] exist Figure 1 In the example of , the client of parallel computing server 104 is a program 110 (e.g., written in Python) interpreted by an interpreter 112 (e.g., a Python interpreter) in client computer 102. If parallel computing server 104 is an Arkouda server, program 110 is an Arkouda client.

[0033] Arkouda supports high performance computing (HPC) in the context of interactive sessions, where humans (e.g. Figure 1 Programmers 108) can participate as part of the data manipulation. Arkouda provides a front end that supports interactive programming, such as a front end that supports Python programming (including a user interface). Programmers 108 can use the front end to develop programs 110. An example of a platform that provides a front end for Python programming or other types of interactive programming is the Jupyter platform. In other examples, other platforms that provide a front end that supports interactive programming can be used.

[0034] According to some embodiments of the present disclosure, the parallel computing server 104 includes a FAM dataset storage manager 130. The FAM dataset storage manager 130 allows a programmer (such as a programmer 108 at a client computer 102) to interact with data residing in the memory of a computer node (such as the local memory 122-1 to 122-N of the corresponding computer nodes 120-1 to 120-N). The "local memory" of a computer node can refer to a memory that is part of the computer node or can be accessed by the computer node through a relatively high-speed link, and its access latency is lower than the access latency of accessing data from the FAM 106 through the interconnect 124. The FAM dataset storage manager 130 allows a programmer (such as a programmer 108) in an interactive programming session to employ the FAM 106 in operations applied to data sets that are too large to fit into the local memory of a cluster of computer nodes. Examples of operations include high performance computing (HPC) operations that are part of computationally demanding workloads, such as workloads associated with artificial intelligence (AI), machine learning, data analysis, or other operations applied to relatively large amounts of data or involving complex calculations. The use of FAM allows programmers to use resources more efficiently when executing HPC workloads such as Figure 1 The resources of the computer nodes 120-1 to 120-N.

[0035] A programming interface 118 (or more specifically, an application programming interface or API) may be provided between a front end such as client computer 102 and a computing back end such as parallel computing server 104 that may perform operations initiated by a client (e.g., program 110) across multiple computer nodes. In some examples, programming interface 118 is an Arkouda programming interface.

[0036] When code lines 114 of program 110 are processed by interpreter 112 , corresponding commands 116 are sent by client computer 102 to parallel computing server 104 via programming interface 118 for triggering operations across computer nodes 120 - 1 to 120 -N.

[0037] A command issued by client computer 102 (or a set of commands from client computer 102) may cause parallel computing server 104 to perform operations across computer nodes. Parallel computing server 104 may distribute operations across computer nodes to perform operations in parallel.

[0038] According to some embodiments of the present disclosure, the FAM data set storage manager 130 running in the parallel computing server 104 on the computer nodes 120-1 to 120-N can process the data stored in the FAM 106. The data in the FAM 106 can be copied (such as in response to input from the programmer 108 in the interactive programming session) to the local memory 122-1 to 122-N of the corresponding computer nodes 120-1 to 120-N. The input can be in the form of code lines 114 added by the programmer 108 to the program 10.

[0039] Examples of operations that may be invoked by commands 116 issued based on lines of code 114 in program 110 include data filtering operations, data scattering operations, data aggregation operations, data sorting operations, or other types of operations applied to data in FAM 106 .

[0040] In some examples, the FAM dataset storage manager 130 may also store information representing a workflow that includes operations invoked by a command 116 from a client computer 102. For example, the information representing the workflow includes information about the command that triggers the workflow operation, and the data to which the command is applied. By storing information representing the workflow, the FAM dataset storage manager 130 supports the functions of pausing and resuming, and recovering from failures. The functions of pausing and resuming refer to the ability of the FAM dataset storage manager 130 to pause a workflow (or an operation in a workflow) and to use the information representing the paused workflow to recover from the last known state of the workflow (or operation). The function of recovering from a failure refers to the ability of the FAM dataset storage manager 130 to recover from a workflow (or an operation in a workflow) when the workflow (or an operation in a workflow) fails.

[0041] The FAM dataset storage manager 130 in the parallel computing server 104 acts as a client (referred to as a "FAM client") of the FAM 106. The FAM dataset storage manager 130 can access the FAM 106 using a FAM programming interface such as the FAM API. The FAM programming interface includes functions (also referred to as routines or methods) that can be called by processes running in the computer nodes 120-1 to 120-N (including the processes of the parallel computing server 104) to manage access to the FAM 106. In some examples, the FAM programming interface includes the OpenFAM API.

[0042] Examples of functions of the FAM programming interface may include a memory allocation function, which is called by a process to allocate a memory area in FAM 106. Figure 1In the example of , memory regions 126-1 through 126-M have been allocated, where M ≥ 1. A "memory region" of FAM 106 may refer to any portion of FAM 106. A memory region may include a contiguous portion of FAM 106, or alternatively, a memory region may include a collection of non-contiguous portions of FAM 106. A memory region may span storage regions of multiple memory devices (or may include storage regions of only one memory device).

[0043] The allocated memory area may be accessed by multiple computer nodes ( 120 - 1 to 120 -N) (the memory area may be shared by multiple computer nodes). Multiple memory areas may be allocated in the FAM 106 .

[0044] The FAM programming interface also includes a memory de-allocation function for de-allocating memory areas in FAM 106. Other functions of the FAM programming interface include a put function for writing data, a get function for reading data, a gather function for gathering data, a scatter function for scattering data, a copy function for copying data, functions for performing atomic operations (e.g., fetch and add, compare and exchange or other atomic operations) or other types of functions. The FAM programming interface also includes a mapping function for mapping data items in FAM106 to the virtual address space of a process (e.g., a process of the FAM data set storage manager 130). The FAM programming interface also includes a de-mapping function for de-mapping data items from the virtual address space of a process.

[0045] Because FAM 106 is associated with higher latency than the local memory 122-1 to 122-N of computer nodes 120-1 to 120-N, the FAM dataset storage manager 130 copies data from FAM 106 to the local memory so that more efficient data processing can be performed through operations triggered by commands 116 issued by program 110 (client) to parallel computing server 104.

[0046] The data in FAM 106 is distributed across local memories 122-1 through 122-N (or subsets of local memories 122-1 through 122-N). As an example, FAM dataset storage manager 130 (in its role as a FAM client) can issue calls to aggregate functions through the FAM programming interface to gather values ​​from one or more data batches 128 stored in memory area 126-1 (or data batches of another memory area) into one or more local memories of computer nodes 120-1 through 120-N. A "batch of data" stored in FAM 106 can refer to a designated unit of data or metadata that can be accessed individually by FAM dataset storage manager 130, such as a batch of data stored in memory area 126-1.

[0047] Data batch 128 contains data and metadata. The data in data batch 128 includes data generated by a user, program, machine, or any other entity. Metadata includes information about the data. Examples of metadata will be discussed further below.

[0048] The FAM dataset storage manager 130 organizes the data batches 128 into FAM datasets 132-1 to 132-P, where P≥1, which are stored in the local memories 122-1 to 122-N of the computer nodes 120-1 to 120-N. The data of the FAM dataset can be distributed across multiple local memories. The FAM dataset represents a logically integrated view of a collection of related data batches accumulated in the FAM 106 over time. The FAM dataset 132-1 contains data for related data batches 128 in the memory area 126-1, while the FAM 132-P contains data for other related data batches 128 in the memory area 126-1. A data batch is "related" if it is the subject of one or more operations requested to be performed by the parallel computing server 104.

[0049] FAM data set 132-1 to 132-P is included in FAM data set storage 136, which stores FAM data sets including data of specific memory areas. For example, FAM data set storage 136 includes FAM data sets in memory area 126-1 of FAM 106. Other FAM data set storages (not shown) containing data of other memory areas of FAM 106 can be provided. FAM data set storage is a logical storage through which a programmer (e.g., programmer 108) can view data to be applied to it. Operations called on a computer node in response to a command 116 from a client (program 110) are also applied to the data of the FAM data set storage, which can be distributed across the local memory of a computer node.

[0050] The arrangement of the FAM data set can be similar to an Arkouda DataFrame, which refers to a data structure that arranges data into multiple columns. Each column of the DataFrame contains an Arkouda parallel distributed array object, called a pdarray object (or simply "pdarray"). A programmer (e.g., 108) can use pdarray objects arranged as columns of an ArkoudaDataFrame. In other words, a code line 114 in a program 110 developed by a programmer can include a reference to a pdarray object.

[0051] To support operations on data in a FAM data set (which may be in the form of a data array such as a pdarray), the programming language of program 110 (e.g., Python) may include a data array class (e.g., a pdarray class) that may be included in a program statement that triggers operations on data arrays in code line 114. An instance of a data array class represents an in-memory data array (e.g., a pdarray) of a FAM data set stored in a local memory of a computer node.

[0052] In some examples, instead of a single FAM data set storage manager, multiple FAM data set storage managers can cooperate in operation to operate different parts of a FAM data set. For example, a first FAM data set storage manager executed on a first subset of computer nodes can apply the operation to a first part of a FAM data set, and a second FAM data set storage manager executed on a second subset of computer nodes can apply the operation to a second part of a FAM data set. These FAM data set storage managers can operate concurrently, thereby improving the parallelism of the execution operation. In addition, depending on the characteristics of the operation to be performed, more (or less) FAM data set storage managers can contribute to the execution of a given operation. For example, a scatter or gather operation can be efficiently processed by multiple FAM data set storage managers. However, if the FAM data set is not too large (e.g., less than a specified threshold), it may be more effective to use a single FAM data set storage manager rather than multiple FAM data set storage managers to apply a sorting operation to the FAM data set.

[0053] Techniques according to some examples of the present disclosure may determine how many computer nodes 120-1 through 120-N to assign to a programmer (e.g., programmer 108) for an interactive programming session, such as by considering the amount of data the programmer plans to process and what operations they plan to perform. For example, if the programmer plans to perform a sort operation on a very large amount of data, the operation may be performed faster using a large number of computer nodes with a large amount of memory than using a small number of computer nodes with a small amount of memory.

[0054] Techniques according to some examples of the present disclosure may determine which computer resources to allocate, such as by considering the characteristics of the operations that the programmer intends to perform. For example, some operations may execute faster on a graphics processing unit (GPU) than on a central processing unit (CPU), so allocating a computer node equipped with a GPU may result in faster execution time.

[0055] Example FAM dataset and operations on the FAM dataset

[0056] According to some examples of the present disclosure, pdarray objects are arranged as columns in a FAM dataset. Figure 2 An example process 200 is shown involving a FAM data set 200 that includes memory regions from a FAM 106 (e.g., Figure 1 126-1 in the FAM dataset 200). The first part 202-1 of the FAM dataset 200 contains data from the first data batch, and the second part 202-2 of the FAM dataset 200 contains data from the second data batch. A given column of the FAM dataset 200 can be a pdarray object containing data values ​​of corresponding attributes. The FAM dataset 200 includes five columns (pdarray objects) representing attributes A, B, C, D, and E. The FAM dataset 200 also includes an index 204. In the example, the index 204 has values ​​0, 1, 2, ..., 20, 21, ... which refer to the corresponding rows of the FAM dataset 200. In other examples, the index 204 may include other values ​​to identify the corresponding rows of the FAM dataset 200.

[0057] Typically, a FAM dataset includes an index (e.g. Figure 2 204 in ) and a set of columns identified by a column name (for example, Figure 2 A, B, C, D, and E in the FAM dataset). The index of a FAM data set includes data values ​​that reference rows of the FAM data set. The index can be used to retrieve data rows from the FAM data set.

[0058] The pdarray object represents a collection of data arrays in physical memory (data arrays stored in local memory 122-1 to 122-N). Figure 2 In the example, pdarray objects containing data values ​​for attribute A may be distributed across the local memory of the computer nodes, pdarray objects containing data values ​​for attribute B may be distributed across the local memory of the computer nodes, pdarray objects containing data values ​​for attribute C may be distributed across the local memory of the computer nodes, pdarray objects containing data values ​​for attribute D may be distributed across the local memory of the computer nodes, and pdarray objects containing data values ​​for attribute E may be distributed across the local memory of the computer nodes.

[0059] The programmer can apply various operations to the pdarray objects contained in the FAM data set. The operations applied to the pdarray objects produce new derived pdarray objects. If the parallel computing server 104 is an Arkouda server, the Arkouda server uses Chapel to distribute the operations across multiple computer nodes for parallel execution.

[0060] According to some examples of the present disclosure, FAM datasets may differ from Arkouda DataFrames in several aspects. First, the index and column data of a FAM dataset may include data values ​​from more than one batch of data in FAM 106. The second difference is that the FAM dataset storage manager 130 supports the creation of derived indexes and columns generated by applying operations to existing indexes and columns of a FAM dataset. Operations that generate new indexes (such as filtering operations or sorting operations) can generate derived FAM datasets. For example, a filtering operation or sorting operation applied to a first FAM dataset (which has a first index) can result in the generation of a second FAM dataset having a second index generated by filtering or sorting the rows of the first FAM dataset. The filtering operation or sorting operation changes the order of the rows of the first FAM dataset and thus results in the derivation of a new index. In contrast, operations (e.g., aggregation, dispersion, subtraction, addition) that use the existing data order to generate new column data will generate derived columns. Such operations do not change the order of the rows of the existing FAM dataset.

[0061] Figure 2 An example of a filter operation 206 is shown for filtering the FAM dataset 200 based on the following filter condition: E>10,000. The filter operation 206 selects rows of the FAM dataset 200 where the attribute E is greater than 10,000. The derived index 210 is the result of the filter operation. Figure 2 In the example of , the rows of FAM data set 200 that satisfy the filter condition are the rows referenced by index values ​​3 and 6. As a result, the derived index 210 produced by the filter operation 206 includes index values ​​3 and 6. Note that because the filter operation 206 has filtered out index values ​​before index value 3, between index values ​​3 and 6, and after index value 6, the order of the index values ​​has changed (due to the omission of index values ​​present in index 204), and the index output by the filter operation 206 is a derived index.

[0062] exist Figure 2 In the example of , the filter operation 206 does not generate output columns. In other words, the filter operation 206 only outputs the derived index 210, but does not output the columns of the rows referenced by the index values ​​of the derived index 210.

[0063] Figure 3 shows how to use the derived index 310 (e.g., Figure 2 The derived index 210) extracts data from an existing FAM dataset 300 (e.g., Figure 2 An example of clustering the existing FAM dataset 200) into a new FAM dataset 302 (based on applying the clustering operation 306). The existing FAM dataset 300 includes a FAM dataset 302 indexed by an index 304 (similar to Figure 2 The index value of index 204) references columns A, B, C, D, and E. Figure 3 The operation aggregates the attributes C and D of the rows in the existing FAM data set 300 referenced by the derived index 310 into the new FAM data set 302. The aggregated attributes C and D are derived columns produced by the aggregate operation 306. Note that the aggregate operation 306 does not change the order of the derived index 310.

[0064] Figure 4 The application to a FAM dataset 402 (e.g., Figure 3 406 to derive a new column 404 that is added to columns C and D of the FAM dataset 402. The FAM dataset with the newly derived column 404 is shown in FIG. Figure 4 It is represented as 402'.

[0065] Generally, according to some examples of the present disclosure, the FAM dataset storage manager 130 can provide the following functions. The FAM dataset storage manager 130 stores data in the FAM area (e.g., Figure 1 126-1) in a batch of data (for example, Figure 1 128) are organized into FAM datasets (e.g. Figure 1 The FAM dataset storage manager 130 supports the creation of derived indexes and columns based on applying operations to existing FAM datasets. The result of the operation can include a derived index (if the operation changes the order of the data) or a derived column (if the operation does not change the order of the data).

[0066] Example arrangement with multiple programs running on different subsets of computer nodes

[0067] Figure 5 is a block diagram of multiple programs that have shared access to FAM 502. These programs include ingestion program 504, first parallel compute server 506, and second parallel compute server 508. For example, first parallel compute server 506 and second parallel compute server 508 may be Arkouda servers.

[0068] Ingestion program 504 is executed on a first subset 514 of computer nodes. A first parallel compute server 506 is executed on a second subset 516 of computer nodes. A second parallel compute server 508 is executed on a third subset 518 of computer nodes. In some examples, first subset 514, second subset 516, and third subset 518 of computer nodes are disjoint subsets that do not share any computer nodes. In further examples, first subset 514, second subset 516, and third subset 518 of computer nodes may share one or more computer nodes.

[0069] The computer nodes include respective local storage. For example, the first subset 514 of computer nodes includes respective local storage 534, the second subset 516 of computer nodes includes respective local storage 536, and the third subset 518 of computer nodes includes respective local storage 538.

[0070] Ingestion program 504 receives data from various data sources 512 and ingests (writes) the received data into FAM 502, such as corresponding batches of data in a memory area of ​​FAM 502. Ingestion program 504 can be, for example, a Chapel program or a different type of program. Data sources 512 can include any one or some combination of the following: a database, a web resource, a program executed on a remote computer, a machine, or any other entity capable of generating or outputting or forwarding data.

[0071] Ingestion program 504 incrementally writes data to FAM 502. Incrementally writing data to FAM 502 means writing portions of data as they are received from data source 512. The data written to FAM 502 is retrieved by FAM data set storage manager 510 to a FAM data set (e.g., Figure 1 During the interactive programming session, the program 522 may access the FAM data set in the local memory 536 of the second subset 516 of computer nodes.

[0072] Based on the operation triggered by the program 522, the FAM dataset storage manager 510 retrieves data from the data batch in the FAM 502 into the FAM dataset across the local storage 536 of the computer nodes. In some examples, the data ingested by the ingestion program 504 can be stored in a one-dimensional data array in the FAM 106. The one-dimensional data array forms the data batch. In other examples, the data can be ingested into multi-dimensional arrays in the data batch. In examples where the data batch includes a one-dimensional data array, the FAM dataset storage manager 510 retrieves the data from the one-dimensional data array into the FAM dataset (such as Figure 2 In multiple columns of the FAM dataset 200).

[0073] A new FAM data set may be derived or an existing FAM data set may be updated (based on generating derived columns or derived indexes) based on the operations triggered by program 522. Further operations may be triggered on the derived or updated FAM data set by a programmer at client computer 520 or another programmer at another client computer that may interact with a FAM data set storage manager of another parallel computing server (not shown).

[0074] FAM datasets (including existing FAM datasets and derived FAM datasets) can be shared by multiple programmers. The sharing of FAM datasets is based on using a common FAM (FAM 502) to store data provided to the FAM datasets. The FAM dataset storage manager 510 writes the data of the derived FAM dataset to one or more data batches in FAM 502. The data of the derived FAM dataset can be used by other entities, including programmers or any other entity. In some examples, the FAM dataset storage manager (e.g., Figure 1 130 or Figure 5 510) may write metadata to the data batch to indicate whether the data batch (or a portion of the data batch) has been published for use by another entity, such as another FAM dataset storage manager, another programmer participating in another interactive programming session, Figure 5 The metadata may identify which entities are allowed to access the data in the data batch.

[0075] The metadata may identify which batches of data in the FAM 106 hold data for which columns and indexes of the corresponding FAM dataset. For example, the metadata may identify that a batch of data with identifiers IDx and IDy contains Figure 3 The metadata for columns C and D of the FAM dataset 302 is shown in FIG.

[0076] Before an entity accesses data in a data batch containing data of a derived FAM data set, the entity checks the metadata of the data batch to determine whether the entity is allowed access - if not, the entity does not access the data in the data batch. The metadata may also indicate which portion of the data batch (e.g., which derived indexes or derived columns) is accessible. The entity will only access the indicated portion of the data batch.

[0077] The first parallel computing server 506 includes a FAM data set storage manager 510 (which may be similar to Figure 1 The client computer 520 (which may be similar to Figure 1 A programmer at a client computer 102 of the embodiment of the present invention can interact with the FAM data set storage manager 510 to initiate operations on data in the FAM 502. The operations initiated are operations requested by commands of lines of code in a program 522 being written at the client computer 520. The lines of code in the program 522 are executed during an interactive session with the first parallel computing server 506.

[0078] The second parallel computing server 508 includes an automatic data updater 524 that incrementally updates the data in the FAM 502. For example, the data updater 524 can incrementally update the derived FAM data set. The incremental updates performed by the data updater will be discussed further below.

[0079] exist Figure 5 In the example of , FAM 502 includes multiple memory servers 540-1, 540-2, ..., 540-Q, where Q≥2. Memory servers 540-i (i=1 to Q) include processing resources (not shown) and corresponding memories. For example, memory server 540-1 includes memory 542-1, memory server 540-2 includes memory 542-2, and memory server 540-Q includes memory 542-Q. The processing resources of memory server 540-i include one or more processors. Computer nodes (in subsets 514, 516, and 518) can access memory servers 540-1 to 540-Q via interconnect 550.

[0080] Example module to support storage management of FAM datasets

[0081] Figure 6 Support for FAM dataset storage manager 602 (which may be Figure 1 FAM data set storage manager 130 in or Figure 5 Various example modules for FAM data set storage management performed by 508 in the embodiment of the present invention. A "module" may refer to a machine-readable instruction that performs a specific task.

[0082] The FAM dataset storage manager 602 is a parallel computing server 604 (which may be Figure 1 The parallel computing server 104 or Figure 5 510). The client computer (e.g., Figure 1 102 or Figure 5 520) provides a parallel computing client 606 (which may be Figure 1 Program 110 or Figure 5 522) to issue commands to the parallel computing server 604.

[0083] The FAM dataset storage manager 602 uses the built-in capabilities of programming languages ​​such as Chapel to distribute operations invoked by the parallel computing client 606 across multiple computer nodes for parallel execution. The FAM dataset storage manager 602 operates as a server (e.g., Arkouda server) relative to the parallel computing client 606 and as a client (e.g., FAM client) relative to the FAM 612.

[0084] In some examples, the parallel computing server 604 may also include a FAM module 608 and a FAM array storage module 610. The FAM module 608 enables the parallel computing server 604 to call FAM operations to access the FAM 612. The FAM module 608 presents a FAM programming interface to the FAM dataset storage manager 602. The FAM dataset storage manager 602 accesses the FAM programming interface to perform various operations for the FAM 612. In some examples, the FAM programming interface presented by the FAM module 608 includes an API having functions that can be called by the FAM dataset storage manager 602 executed on the computer node to manage or access the FAM 612. An example of such an API is an OpenFAM API that supports OpenFAM operations.

[0085] The FAM data set storage manager 602 can be stored across multiple computer nodes (such as Figure 1 Computer nodes 120-1 to 120-N or Figure 5 More generally, the FAM data set storage manager 602 may be executed across multiple processing sites. As described above, a processing site refers to any different computing partition, such as a computer node or a portion of a computer node.

[0086] The FAM dataset storage manager 602 is executed across multiple processing sites by running multiple instances of the FAM dataset storage manager 602 in the corresponding processing sites. Each instance of the FAM dataset storage manager 602 is a process entity running in a corresponding processing site.

[0087] In some examples, because the FAM module 608 is not responsible for managing the distribution of operations across processing sites, the instance of the FAM dataset storage manager 602 running in a given processing site ensures that the addresses (e.g., virtual addresses) of remote memory access (RMA) operations issued from the instance of the FAM dataset storage manager 602 to the FAM 612 are addresses within the address space of the given processing site. Since the first instance of the FAM dataset storage manager 602 is restricted to use the address space of the first processing site when accessing the FAM 612, the first instance of the FAM dataset storage manager 602 will not be able to write data in the address space of the second processing site to the FAM 612.

[0088] The FAM array storage module 610 handles data distribution across processing sites (such as to local storage of computer nodes). As described above, the columns of the FAM data set (such as Figure 2Any of the columns A, B, C, D, and E of the illustrated FAM data set 200 may be arranged as a pdarray object. The FAM array storage module 610 is capable of distributing pdarray objects across multiple processing sites.

[0089] The FAM array storage module 610 is capable of converting operations on pdarray objects into FAM-specific accesses of FAM 612. When an operation is called to be performed across multiple processing sites, the multiple processing sites are allocated corresponding portions of data in FAM 106 for processing. In other words, a first portion of the operation performed on a first processing site is allocated a first portion of data in FAM 106 for processing, and a second portion of the operation performed on a second processing site is allocated a second portion of data in FAM 106 for processing. As an example, the FAM array storage module 610 can evenly divide the data in FAM 106 for processing across multiple processing sites. For example, if the data in FAM 612 has 100 data elements to be distributed across two processing sites, the FAM array storage module 610 can place 50 elements in each processing site.

[0090] In some examples, if there are multiple memory regions (e.g., Figure 1 126-1 to 126-M), an instance of the FAM array storage module 610 can be provided for each memory area.

[0091] The FAM data set storage manager 602, FAM module 608, and FAM array storage module 610 allow a programmer (e.g., Figure 1 108) to create, access, and manipulate data stored in the FAM without having to be familiar with the details associated with accessing and storing data in the FAM 106. The logistics of moving data between the local storage of a computer node and the FAM are handled by the FAM module 608 and the FAM array storage module 610, and the FAM data set storage manager 602 manages the distribution of data operations across multiple processing sites.

[0092] Incremental updates of derived FAM datasets

[0093] refer to Figure 7 In some examples, data updater 702 (e.g., which may be Figure 5The data updater 524 of the embodiment of the present invention is capable of incrementally updating a derived FAM data set including one or both of a derived index and a derived column. In order to support incremental processing of a derived FAM data set as an update to the underlying base data (the data of an existing FAM data set from which the derived FAM data set is generated), the data updater 702 can store derivative relationship information 704 specifying the derivative relationship between the base FAM data set and the derived FAM data set. A "base" FAM data set is a FAM data set to which a derivative operation is applied to generate a result in the form of a derived FAM data set. A "derivation operation" is an operation that derives a FAM data set including a new index or a new column.

[0094] Note that the base FAM dataset may comprise a FAM dataset directly populated with data from the FAM, or alternatively, the base FAM dataset may be another derived FAM dataset formed by another derivation operation.

[0095] The derivation relationship information 704 specifies which base FAM data set 706 is used to generate a derived FAM data set 708 based on applying a derivation operation 710 to the base FAM data set 706. The derivation relationship can specify a series of derivation operations that can be applied to multiple levels of FAM data sets. For example, a first derivation operation is applied to a base FAM data set to form a first derived FAM data set, and a second derivation operation is applied to the first derived FAM data set to form a second derived FAM data set.

[0096] The derived relationship information 704 may be stored in a memory 712, which may be Figure 1 The local memory 122-1 to 122-N or Figure 6 In addition, the data updater 702 can distinguish between order-preserving derivative operations and order-destroying derivative operations that create derived FAM data sets. As discussed further above, filter operations and sort operations are examples of operations that change the order of rows of an existing FAM data set; these types of operations are order-destroying derivative operations. In contrast, gather, scatter, subtract, and add operations are examples of operations that do not change the order of an existing FAM data set; these types of operations are order-preserving operations.

[0097] Tracking may be accomplished by storing, by the data updater 702, an order indicator 714 (eg, a flag or parameter) that specifies whether the derivation operation is an order-preserving derivation operation or an order-destroying derivation operation.

[0098] Order-preserving derivative operations are performed on a per-batch basis; in other words, ingesting new data into a data batch corresponding to a FAM dataset does not affect the results of the previously computed derived FAM dataset. For example, updating a derived index or derived column in response to the ingestion of new data into a data batch is straightforward, as the new data can be processed and simply appended to the existing results in the derived FAM dataset.

[0099] On the other hand, since a de-ordering operation such as a sorting operation will scatter data across all results in a derived FAM dataset, new data in an ingested data batch cannot be simply appended to the results in the derived FAM dataset, because the new data must be re-ordered together with the data in the results in the derived FAM dataset. More generally, when the ingested new data affects the results of a derived FAM dataset produced by a de-ordering operation, the computation of the de-ordering operation will have to be applied jointly to the new data as well as the data in the results in the derived FAM dataset.

[0100] Using an order indicator 714 that specifies whether a derive operation 710 that produces a derived FAM data set 708 is an order-preserving derive operation or an order-destroying derive operation, the data updater 702 can intelligently determine, as part of an incremental update of the derived FAM data set 708, whether (1) the derive operation 710 can be applied to the newly ingested data and the result appended to the derived FAM data set 708, or (2) the derive operation 710 is to be applied jointly to the newly ingested data and the data in the result of the derived FAM data set 708. In response to the data updater 702 receiving an indication from the FAM that new data has been ingested, the incremental update is performed.

[0101] The ability to incrementally update a derived FAM data set allows a programmer in an interactive programming session to apply operations to large data sets whose total size may exceed the combined storage capacity of the local memories of the computer nodes. For such large data sets, segments of the large data set may be incrementally ingested into the FAM 106, and incremental updates may be applied to the derived FAM data set, wherein the segments of the large data set incrementally ingested into the FAM 106 may fit within the combined storage capacity of the local memories of the computer nodes.

[0102] As the derived FAM data set is incrementally updated, the corresponding metadata associated with the derived FAM data set is also updated by the data updater 702 .

[0103] In some examples, incremental update can be set to be automatically performed by data updater 702. For example, automatic update indicator 716 can be stored in memory 712 to indicate whether to apply incremental update in response to changes in the basic FAM data set. If automatic update indicator 716 is set to a first value, then when changes occur in the basic FAM data set, data updater 702 will automatically apply incremental update. On the other hand, if automatic update indicator 716 is set to a different second value, then when changes occur in the basic FAM data set, data updater 702 will not apply incremental update. For example, automatic update indicator 716 can be set by a programmer.

[0104] Other Examples

[0105] Figure 8 is a flow chart of a process 800 according to some examples of the present disclosure. The process 800 may be performed by a parallel computing server, such as Figure 1 104 of 104 Figure 5 506 or Figure 6 604 of them.

[0106] Process 800 includes receiving (at 802), by a parallel computing server executing in a system including a plurality of computer nodes, commands based on program code of a program being developed in an interactive programming session. An "interactive programming session" refers to a session associated with developing a program in which lines of code of the program are executed as the program is developed.

[0107] Process 800 includes distributing (at 804) data items from a network attached storage to a distributed data object comprising data in node storage of a plurality of computer nodes. An example of a network attached storage is Figure 1 FAM 106 Figure 5 FAM 502 or Figure 6 FAM 612 in. "Data item" refers to any unit of data, such as in the form of a batch of data or any other unit of data. "Distributed data object" refers to a data object having multiple parts stored in different node memories. "Node memory" refers to the memory of a computer node, where the memory may be part of the computer node, or may be independent but accessible to the computer node.

[0108] Process 800 includes performing (at 806) an operation specified by a command on a distributed data object by a data set manager executing in the system, the operation being performed in parallel on a plurality of computer nodes. The operation performed in parallel on the computer nodes refers to applying the operation on different portions of the distributed data object on different respective computer nodes. The data set manager refers to machine-readable instructions that manage retrieval of data from a network attached storage, distribution of the distributed data object across computer nodes, and application of the operation to the distributed data object to generate derived data.

[0109] Process 800 includes generating (at 808), by a data set manager in the system, derived data generated by operations on distributed data objects, which derived data is accessible to a programmer in an interactive programming session. "Derived data" refers to data generated by calculations performed on other data. Derived data is "accessible" to the programmer in that the programmer is able to add lines of code to operate on the derived data.

[0110] In some examples, the data item to which the operation is applied is a portion of a data array stored in a network attached memory, the total size of the data array being greater than a memory capacity of any node memory of the plurality of computer nodes.

[0111] In some examples, process 800 includes storing, by the data set manager, the derived data in the network attached storage, and sharing the derived data stored in the network attached storage with another programmer in another interactive programming session.

[0112] In some examples, process 800 includes receiving additional data by an ingestion program executing in the system, storing the additional data as additional data items in the network attached storage by the ingestion program, and incrementally updating the derived data based on the additional data items. Figure 5 Data updater 524 or Figure 7 The data updater 702 in performs incremental updates.

[0113] In some examples, process 800 includes receiving, by the data set manager, an indication that an automatic update of derived data is to be performed. An example of an indication is Figure 7 The automatic update indicator 716 of the derived data is provided. The incremental update of the derived data is responsive to the indicator.

[0114] In some examples, the incremental update is performed by a data updater executing on another computer node than the plurality of computer nodes on which the data set manager executes.

[0115] In some examples, the distributed data object is part of a dataset, and process 800 includes setting, by a dataset manager, metadata associated with the dataset indicating that the dataset is published for access by another dataset manager associated with another programmer in another interactive programming session.

[0116] In some examples, the distributed data object is part of a data set, and the derived data includes indices of rows of the data set that satisfy a condition. For example, the condition may be a condition that is determined by Figure 2 The filtering operation 206 specifies the filtering conditions.

[0117] In some examples, the distributed data object is part of a data set, and the derived data includes derived column data of the data set, the derived column data being generated by an operation on one or more columns of the data set. An example of derived column data is Figure 4 The derived column in 404.

[0118] In some examples, the program code of the interactive programming session includes another command that includes a class that references a distributed data object. An example of such a class is the data array class mentioned further above.

[0119] In some examples, process 800 includes presenting a programming interface that includes functions that can be accessed by a data set manager to access data in a network attached storage. For example, the programming interface can be provided by Figure 6 The FAM module 608 presents the API.

[0120] In some examples, process 800 includes presenting, by the data set manager, data of the data item to the programmer in a column format. For example, the data of the data item is included in a plurality of columns of the FAM data set.

[0121] Fig. 9 is a block diagram of a system 900 including a plurality of computer nodes 902 with corresponding node memory 904. The system 900 also includes a storage medium 906 storing machine-readable instructions executable on at least one of the plurality of computer nodes 902 to perform various tasks.

[0122] The machine readable instructions include command receiving instructions 908 for receiving commands based on program code of a program being developed in an interactive programming session. An "interactive programming session" refers to a session associated with developing a program in which lines of code of the program are executed as the program is developed.

[0123] The machine-readable instructions include data retrieval instructions 910 for retrieving data from a network attached storage through an interconnect to a plurality of computer nodes based on a command. The machine-readable instructions include retrieve data storage instructions 912 for storing the retrieved data in a distributed data object distributed across the node memories. A "distributed data object" refers to a data object having multiple portions stored in different node memories.

[0124] The machine-readable instructions include operation performance instructions 914 for the data set manager to perform the operation specified by the command on the distributed data object, the operation being performed in parallel on multiple computer nodes. The operation performed in parallel on the computer nodes means applying the operation to different parts of the distributed data object on different computer nodes.

[0125] The machine-readable instructions include derived data generation instructions 916 for generating, by the data set manager, derived data generated by operations on distributed data objects, which derived data is accessible to a programmer in an interactive programming session. "Derived data" refers to data generated by calculations applied to other data. Derived data is "accessible" to the programmer in that the programmer is able to add lines of code to operate on the derived data.

[0126] Fig.10 is a block diagram of a non-transitory machine-readable or computer-readable storage medium 1000 storing machine-readable instructions executable on the system to perform various tasks. The system may include multiple computer nodes.

[0127] The machine-readable instructions include data ingest instructions 1002 for ingesting data from a data source into a network attached storage. "Ingesting" data may refer to receiving data and writing the data to a target storage.

[0128] The machine-readable instructions include command receiving instructions 1004 for receiving commands based on the interpretation of program code of a program being developed on a client computer in an interactive programming session. "Interpreting" program code means converting the program code into a form executable by a machine without compiling the program code.

[0129] The machine-readable instructions include data retrieval instructions 1006 for retrieving data from the network attached storage through the interconnect to the plurality of computer nodes based on the command. The machine-readable instructions include retrieve data storage instructions 1008 for storing the retrieved data in a distributed data object of the node storage distributed across the plurality of computer nodes.

[0130] The machine-readable instructions include operation performance instructions 1010 for the data set manager to perform an operation specified by the command on the distributed data object, the operation being performed in parallel on multiple computer nodes. The operation performed in parallel on the computer nodes refers to applying the operation to different parts of the distributed data object on different computer nodes.

[0131] The machine-readable instructions include derived data generation instructions 1012 for generating, by the dataset manager, derived data generated by operations on the distributed data objects, the derived data being accessible to a programmer in an interactive programming session.

[0132] The machine-readable instructions include derived data write instructions 1014 for writing derived data to the network attached storage, where the derived data in the network attached storage is accessible to another programmer. For example, the other programmer may use another data set manager to access the derived data.

[0133] Storage media (e.g. Fig. 9 906 or Fig.10 1000) may include any one or some combination of the following: semiconductor memory devices such as DRAM or SRAM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; disks such as fixed disks, floppy disks, and removable disks; another magnetic medium including tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or another type of storage device. Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or may be provided on multiple computer-readable or machine-readable storage media distributed in a large system that may have multiple nodes. Such computer-readable or machine-readable storage media are considered to be part of an article (or product). An article or product may refer to any manufactured single component or multiple components. One or more storage media may be located in a machine that runs machine-readable instructions, or at a remote site from which machine-readable instructions may be downloaded over a network for execution.

[0134] In the present disclosure, the use of the terms "a", "an" or "the" is also intended to include plural forms, unless the context clearly indicates otherwise. In addition, the terms "comprises", "comprising", "including", "containing", "having" or "having" when used in the present disclosure specify the presence of the elements described, but do not exclude the presence or addition of other elements.

[0135] In the foregoing description, many details are set forth to provide an understanding of the subject matter disclosed herein. However, some of these details may be implemented without them. Other embodiments may include modifications and variations to the above details. The appended claims are intended to cover these modifications and variations.

Claims

1. A method, include: At a system including a plurality of computer nodes in an interactive programming session, receiving commands based on program code of a program being developed in the interactive programming session; distributing, by the system, data items from the network attached storage to a distributed data object, the distributed data object comprising the data in the node storages of the plurality of computer nodes; performing, by a data set manager executing in the system, an operation specified by the command on the distributed data object, the operation being performed in parallel on the plurality of computer nodes; as well as Derived data generated by the operations on the distributed data objects is produced by the data set manager in the system, and the derived data is accessible to a programmer in the interactive programming session.

2. The method according to claim 1, in, The data item to which the operation is applied is a portion of a data array stored in the network attached memory, the total size of the data array being greater than the memory capacity of any node memory of the plurality of computer nodes.

3. The method according to claim 1, further comprising: include: storing, by the data set manager, the derived data in the network attached storage; as well as The derived data stored in the network attached storage is shared with another programmer in another interactive programming session.

4. The method according to claim 1, further comprising: include: receiving additional data via an ingestion program executed in said system; storing, by the ingestion program, the further data as further data items in the network attached storage; and The derived data is incrementally updated based on the further data items.

5. The method according to claim 4, further comprising: include: receiving, by the data set manager, an indication that an automatic update of the derived data is to be performed, Wherein, the incremental update of the derived data is responsive to the indication.

6. The method according to claim 5, in, The incremental update is performed by a data updater executing on another computer node that is different from the plurality of computer nodes on which the data set manager executes.

7. The method according to claim 1, in, The distributed data object is part of a data set, and the method further comprises: Metadata associated with the dataset is set by the dataset manager, the metadata indicating that the dataset is published for access by another dataset manager associated with another programmer in another interactive programming session.

8. The method according to claim 1, in, The distributed data object is part of a data set, and wherein the derived data includes an index of a row of the data set that satisfies a condition.

9. The method according to claim 1, in, The distributed data object is part of a data set, and wherein the derived data includes derived column data of the data set, the derived column data being generated by an operation on one or more columns of the data set.

10. The method according to claim 1, in, The program code of the interactive programming session includes another command including a class referencing the distributed data object.

11. The method according to claim 1, further comprising: include: A programming interface is presented that includes functions accessible by the data set manager to access data in the network attached storage.

12. The method according to claim 1, in, The program code includes a plurality of code lines, and the method further includes: sending a first command for a first line of code in the plurality of lines of code to the system, the first command specifying a first operation related to the network attached storage; and A second command for a second line of code in the plurality of lines of code is sent to the system, the second command specifying a second operation related to the network attached storage.

13. The method according to claim 1, further comprising: include: The data of the data item is presented to the programmer by the data set manager in a column format.

14. A system, include: a plurality of computer nodes, the computer nodes comprising a node memory; a non-transitory storage medium comprising instructions executable on at least one of the plurality of computer nodes, the instructions for: receiving commands based on program code of a program being developed in the interactive programming session; retrieving data from a network attached storage via an interconnect to the plurality of computer nodes based on the command; storing the retrieved data in a distributed data object distributed across the node memories; executing, by the data set manager, the operation specified by the command on the distributed data object, wherein the operation is executed in parallel on the plurality of computer nodes; as well as Derived data generated by operations on the distributed data objects is generated by the data set manager, and the derived data is accessible to a programmer in the interactive programming session.

15. The system according to claim 14, in, The distributed data object is a first distributed data object comprising a first portion of the retrieved data, and wherein the instructions are executable on at least one of the plurality of computer nodes to: storing a second portion of the retrieved data in a second distributed data object distributed across the node memories, The operation is performed by the data set manager on the first distributed data object and the second distributed data object.

16. The system according to claim 15, in, The instructions are executable on at least one of the plurality of computer nodes to: The retrieved data is arranged as columns in a data set stored in the node memory, the columns including a first column containing data of the first distributed data object and a second column containing data of the first distributed data object.

17. The system according to claim 14, in, The dataset manager is a server for the program, and the dataset manager is a client for the network attached storage.

18. The system according to claim 14, in, The instructions are executable on at least one of the plurality of computer nodes to: storing in a memory an indicator of whether the operation is an order-preserving derivation operation or an order-destroying derivation operation; as well as The derived data is incrementally updated using the indicator.

19. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a system to: Ingesting data from data sources into network attached storage; receiving commands based on an interpretation of program code of a program being developed at a client computer in an interactive programming session; retrieving data from the network attached storage via an interconnect to a plurality of computer nodes based on the command; storing the retrieved data in a distributed data object distributed across node memories of the plurality of computer nodes; executing, by the data set manager, the operation specified by the command on the distributed data object, wherein the operation is executed in parallel on the plurality of computer nodes; generating, by the data set manager, derived data generated by operations on the distributed data objects, the derived data being accessible to a programmer in the interactive programming session; as well as The derived data is written to the network attached storage, where the derived data is accessible to another programmer.

20. The non-transitory machine-readable storage medium of claim 19, in, The instructions, when executed, cause the system to: receiving an indication that an automatic update of the derived data is to be performed; Based on the indication, the derived data is incrementally updated based on ingesting additional data into the network attached storage.