Division, Processing, and Protection of Multidimensional Data

Abstract

Techniques for managing multi-dimensional data include providing an original data set that includes data arranged along a plurality of dimensions, each dimension covering a respective original range in dimension units. The technique further includes extracting a plurality of data portions from the original data set, each portion extending over a reduced range of dimension units that is smaller than the original range in at least one dimension, and all of the extracted portions together covering the original range of the original data set in all dimensions.

Description

Technical Field

[0001] Cross - Reference to Related Applications: This invention claims the benefit of U.S. Provisional Application No. 63 / 394,466, filed Aug. 2, 2022. The entire contents and teachings thereof are hereby incorporated by reference in their entirety.

[0002] Statement Regarding Federally Sponsored Research or Development: This invention was made with government support under award number NA22OAR0210591 from the National Oceanic and Atmospheric Administration. The government has certain rights in this invention.

Background Art

[0003] Data processing and protection have undergone dramatic changes due to the widespread availability of inexpensive processors and storage media. Currently, users can choose to process and store data locally or store data on servers, computing clusters, or clouds connected via a network. Additionally, cloud computing options include both public clouds and private clouds.

[0004] Entering the big data era, users desire to store and process a larger number of data objects. For example, it is not uncommon for tabular data, tree - structured data, audio, and / or video data to be in sizes of gigabytes or more. Processing, protecting, and storing such large - volume data objects pose unique challenges.

[0005] A common approach is to divide a large object into multiple parts and store each part on a separate computer. The program can identify byte boundaries within the object and divide the object by generating parts of the same size, or approximately the same size. To perform data processing on a data object stored in a distributed manner, a computer can collect a specific part or group of parts of the original object, perform the desired processing task on the collected parts, and generate a result.

Summary of the Invention

[0006] Unfortunately, the above-described distributed approach can be inefficient. For example, in an approach that divides a large data object evenly or approximately evenly, structural features are ignored and dependencies can arise between different data parts. As a simple example, consider a data object in which tabular data is stored across multiple rows. If the object is divided into equally-sized parts, rows may be truncated in the middle. Therefore, in subsequent queries that involve access to the truncated rows, access may be required to two parts of the data object, one that stores the start part of the row and one that stores the end part of the row. These two parts are typically stored on different computers on the network.

[0007] Continuing with the above example, it may be necessary to further transfer both parts (including both parts of the truncated row) to the requester or some other nodes, where they are recombined to execute the query. These operations are very inefficient because they involve copying large amounts of data over the network.

[0008] In addition to the above, conventional approaches may ignore the content. For example, the divided parts of a data object may lose their relevance to the whole data object. In tabular data, field names may be missing (for example, when only row data is stored). Therefore, in order to extract meaningful data from distributed objects, it may be necessary to send multiple network accesses to different computers to collect all the pieces necessary to complete the desired processing task.

[0009] To address these drawbacks, techniques for managing data objects in a storage cluster include dividing a data object into multiple parts at the boundaries within the data object. This technique further includes converting the parts of the data object into segments that provide units that can be processed individually, and distributing and storing those segments across multiple compute nodes of the storage cluster.

[0010] Advantageously, providing segments as individually processable units means that the workload associated with executing the processing tasks of a data object can be efficiently pushed down to the compute nodes that locally store the segments of that data object. This technique enables true parallel processing where a compute node executes processing tasks only on the segments of the data object stored therein. Also, compared to previous methods, network traffic can be significantly reduced. For example, with high-speed connections to the local storage of compute nodes, the overall efficiency is greatly improved. Furthermore, due to the independence of segments, communication between compute nodes (for example, to resolve dependencies) is almost or completely unnecessary to complete the processing tasks.

[0011] There are specific challenges in the segmentation and processing of multi-dimensional data. First, such data can be of an enormous volume, with datasets typically in the terabyte range and some reaching the petabyte range. For example, considering weather data, the dataset records temperature, humidity, atmospheric pressure, wind speed, precipitation, and other factors. The size of these datasets can become very large, especially when targeting vast geographical areas. Second, multi-dimensional data cannot be naturally segmented for parallel processing. There are various software programs that assist in data extraction, but the extracted data represents only a portion of the original dataset and does not reflect the entire dataset. Therefore, what is needed is a way to render a large dataset as smaller segments that can be processed independently and in parallel to generate information that reflects the entire dataset.

[0012] To at least partially meet this need, improved techniques for managing multi-dimensional data include providing an original dataset that contains data arrayed along multiple dimensions, where each dimension covers its respective original range in dimension units. This technique further includes extracting a plurality of data portions from the original dataset, where each portion extends over a reduced range of dimension units that is smaller than the original range in at least one dimension, and all the extracted portions together cover the original range of the original dataset in all dimensions.

[0013] Advantageously, this improved technique enables rendering a large dataset by splitting it into a plurality of smaller parts. These parts contain the same data as the original dataset but can be processed independently and much more efficiently. In some examples, some parts are stored on each computing node of a storage cluster and can be processed in parallel. By aggregating the results of the parallel processing, a meaningful result that reflects the entire original dataset can be generated, achieving much higher efficiency than when processing the original dataset as one large object.

[0014] One embodiment is directed to a method for managing multi-dimensional data. The method includes providing an original data set that includes data arranged along a plurality of dimensions of an N-dimensional space. Each dimension of the N-dimensional space has a respective original range in dimension units, and the original data set has a data format. The method further includes extracting a plurality of data portions from the original data set. Each portion extends over a reduced range of dimension units in at least one dimension of the N-dimensional space. The extracted portions together cover all of the original ranges of the N-dimensional space. The method further includes rendering the extracted portions into respective segments that provide the data of the extracted portions in the same data format as the original data set.

[0015] An additional embodiment is directed to a computerized apparatus configured and arranged to perform a method for managing multi-dimensional data such as the method described above. Still other embodiments are directed to a computer program product. The computer program product stores instructions that, when executed on a control circuit of a computerized apparatus, cause the computerized apparatus to perform a method for managing multi-dimensional data such as the method described above.

[0016] The above summary is presented for illustrative purposes to enable a reader to readily grasp the features of the examples presented herein. However, the summary is not intended to present essential elements or to limit the embodiments herein in any way. It should be understood that the above features may be combined in any technically meaningful way, and all such combinations, whether or not explicitly specified, are intended to be disclosed herein.

Brief Description of the Drawings

[0017] The above and other features and advantages will become apparent from the following description of particular embodiments shown in the accompanying drawings. In the accompanying drawings, like reference numerals refer to the same or similar elements throughout the different drawings.

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0018] Embodiments of the improved technology will be described below. It should be understood that such embodiments are provided to illustrate specific functions and principles and are not intended to be limiting.

[0019] Techniques for managing data objects within a storage cluster include dividing a data object into a plurality of parts at boundaries within the data object. This technique further includes converting a part of the data object into segments that provide independently processable units, and storing the segments in a distributed manner across a plurality of compute nodes of the storage cluster.

[0020] In the following description, · In Section I, environmental examples and embodiments regarding data division, processing, and protection are shown, · In Section II, application examples of the embodiments of Section I to multi-dimensional data are shown.

[0021] Section I: Data Division, Processing, and Protection This application discloses a plurality of embodiments. One embodiment relates to dividing a data object into parts for distributed storage in a storage cluster. Another embodiment relates to the execution of distributed processing tasks by a storage cluster. Yet another embodiment relates to data protection of data objects stored in a storage cluster. These embodiments may be realized as respective aspects of a single system, as shown and described in the examples below. Alternatively, the embodiments may be implemented independently, and an implementation that supports one embodiment need not support the other embodiments.

[0022] FIG. 1 shows an example environment 100 in which embodiments of an improved technology can be implemented. As shown, gateway 110 is configured to access a plurality of compute nodes 120 of storage cluster 130 via network 140 and function as an interface between storage cluster 130 and clients / users. Network 140 may include a local area network (LAN), a wide area network (WAN), the Internet, or other types of networks or network combinations that support digital communication between computers. Gateway 110 may be a computer or other computing device (e.g., a server, workstation, tablet, smartphone, personal data assistant, gaming console, set-top box, etc.) and may include its own network interface, processor, and memory. In some examples, gateway 110 may be provided as a compute node 120 of storage cluster 130. A plurality of compute nodes 120 (also referred to herein as "nodes") 120-1 to 120-N are shown, but it should be understood that storage cluster 130 may include a large number of nodes 120, such as hundreds or more. Each node 120 includes one or more processors and memory for executing programs, one or more network interfaces (e.g., network interface cards), and one or more persistent storage, such as solid state drives (SSDs), magnetic disk drives, etc. The nodes 120 of storage cluster 130 may be interconnected via network 140, via a dedicated network (e.g., a separate local area network, not shown), or by other means. In this specification, all internal networks of storage cluster 130 are considered part of network 140.

[0023] Preferably, each node 120 has one or more high-speed connections to respective persistent storage. For example, the connection between the nodes 120 and their storage devices (e.g., SSDs) may have a bandwidth that is one digit or more higher than the connection between nodes via the network 140.

[0024] In one example, the storage cluster 130 is configured as an object store and may be compatible with commercially available cloud-based object stores such as AWS (Amazon Web Services)' S3 (Simple Storage Service), Microsoft Azure Data Lake, and / or Google Cloud Storage. In a particular example, the storage cluster 130 is configured as an S3-compatible object store. For this purpose, each node 120 may include an API (Application Program Interface) 122 that enables the node 120 to participate as a member of the object store.

[0025] The cluster 130 may be implemented within a data center that occupies one or more rooms of a building where the nodes 120 are networked together. In other implementations, it may span multiple buildings, and a metro cluster configuration is also achievable.

[0026] In other examples, the storage cluster 130 may be implemented within the cloud service 150, for example, using physical machines or virtual machines provided therein. For example, the entire storage cluster 130 may be fully located within the cloud service 150.

[0027] As yet another example, the cloud service 150 may function as the primary repository of data, and the storage cluster 130 may function as a cache of the cloud service 150. Thus, the storage cluster 130 can store generally accessed data, but usually does not store all the data available from the cloud service 150.

[0028] The implementation is suitable for individuals, small organizations, and / or enterprises and can be provided according to the SaaS (software as a service) model or other models. Embodiments are particularly suitable for the management of large-scale data objects that may have a size of 100 megabytes or more. This feature fits well with big data applications such as those including data lakes. However, it should be understood that the embodiments are not limited to specific users, service models, data sizes, or applications.

[0029] In a processing example, the gateway 110 (which may be part of or independent from the storage cluster 130) accesses one or more data objects 160 managed by the storage cluster 130. The data objects 160 may be present within the cloud service 150, for example, within buckets or blobs, or may be provided from one or more separate sources. For example, the data objects 160 may be generated by real-time activities such as industrial or scientific processes that can generate the data objects 160 as data logs or other records of ongoing activities. The data objects 160 may be represented as files, streams, memory ranges, or other means.

[0030] The data object 160 may be structured according to a specific object type. For example, the data object 160 may be provided as a tabular object such as a CSV (comma-separated values) or a log file, as a tree-based object such as a JSON (JavaScript (registered trademark) Object Notation) or an XML (extensible markup language) document, as a column-oriented object such as an Apache Parquet file, as a video file or stream, as an audio file or stream, or as a collection of images. Although specific types of data are specifically shown and / or described, the embodiments are intended to encompass any type of data, and it should be understood that what is shown and / or described here is merely providing specific examples used to illustrate the operating principle.

[0031] To begin managing data object 160, gateway 110 may scan the data object, for example, starting from the beginning of the data object and proceeding forward. Usually, when gateway 110 first accesses an object, it may not notice the type of the data object, so an initial scan of object 160 may be performed to identify its type. The scan may involve sampling a series of regions of the data object (usually the beginning of the object) and searching for sequences or characters specific to a particular object type. For example, in a CSV file or a log file, a newline character is typically used to indicate the end of a record, and commas, spaces, and other characters can be used to separate adjacent fields. Some data objects contain headers that directly identify the type of the object. For example, a Parquet file begins with a 4-byte header that specifies a so-called "magic number" and uses the code "PAR1" to identify that the file is a Parquet file. Most file types provide a clear indication that can be identified without much effort. However, there are some types that are difficult to identify. If you want to recognize such types that are not easily identifiable, more advanced algorithms, including machine learning and other types of artificial intelligence, may be applied.

[0032] Once gateway 110 identifies the type of data object 160, gateway 110 can begin to split data object 160. For example, gateway 110 can search for boundaries within the data object that provide separators between adjacent processable units of the data object. The exact nature of the boundary may vary depending on the type of the object. For example, a newline character may be used to identify a boundary in a CSV file, while an I-frame (intra-coded picture) may be used in a video file or stream. Depending on the object type, some may use embedded metadata to specify the boundary. For example, a Parquet file contains a footer that indicates the boundary between adjacent row groups.

[0033] The "processable unit" of a data object is an area where independent processing is possible in the sense that it has little dependence on other processable units. By dividing a data object into processable units, efficient parallel processing by the nodes 120 of the storage cluster 130 is promoted.

[0034] Although the division is the first step to facilitate independent processing of the divided parts, it is not necessarily sufficient to achieve optimal performance. For example, the divided parts may lack specific metadata (such as headers, footers, or other content) that maintain the dependence on other parts of the data object 160. Therefore, the gateway 110 preferably performs an additional step of converting the divided parts into segments 170. As an example, the converted segments 170 can be processed as if they were complete self - contained objects of the same type as the data object 160.

[0035] The segments 170 are similar to the parts from which they are generated but are adjusted to reduce or eliminate the degree of dependence on other parts. For example, if the first part of a CSV file contains a header but subsequent parts do not, the gateway 110 can copy the header of the first part to each of the segments 170 formed from the subsequent parts. In this way, each segment 170 has its own header and can be processed like an independent CSV file. Corresponding adjustments can also be made for other object types, and the details of the adjustments depend on the object type. Various examples are shown below.

[0036] Once the segment 170 is thus formed as an independently processable unit of the same type as the data object 160, the gateway 110 can distribute the segment 170 to various nodes 120 of the storage cluster 130, and these nodes 120 store the segment therein, for example, in the persistent storage locally connected to each node 120. To track the location of the segment, the gateway 110 may update the object metadata 112.

[0037] As shown in the enlarged view of FIG. 1, the object metadata 112 includes object-specific information that facilitates the operation of the storage cluster 130. Such object metadata 112 may include, for example, the following elements. · Object ID. It is an object identifier and is preferably unique within the namespace of the storage cluster 130. · Object type. The determined type of the data object 160, such as CSV, JSON, XML, Parquet, etc. · Segment ID. The identifier of the segment 170 created from a part of the object. It is preferably unique within the namespace of the storage cluster 130. · Byte range. The byte range of the data object 160 included in the current segment. It can be expressed as a pair of values (or start byte position and length) specifying the start byte position and the end byte position. · Row range. The range of rows of the data object 160 included in the current segment. It is related to tabular data and other types of data provided in rows. · Feature. Features detected in the segment that may be related to later processing. It may be provided for each segment. Although shown as a single-level structure, the object metadata 112 may be arranged in any suitable way including a hierarchical structure. Also, the scope of the object metadata 112 is not limited to the provided examples. In fact, the object metadata 112 can store any information that facilitates the operation of the storage cluster 130 or processing tasks that can be executed therein.

[0038] In some examples, the object metadata 112 is stored redundantly to increase reliability. For example, the object metadata 112 can be stored on multiple nodes 120 of the storage cluster 130, for example, using a multi-way mirror and / or other RAID (Redundant Array of Independent Disks) or erasure coding techniques. Also, the activities attributed to the gateway 110 herein may be performed by any number of computers, and such computers may include the nodes 120 of the storage cluster 130. For example, a particular node of the storage cluster 130 can be designated as a load balancer, and the workload of the node 120 can be considered when segments 170 are distributed among the nodes of the cluster.

[0039] As further shown in FIG. 1, the compute nodes 120 can store segment metadata 124 that describes the segments 170 stored by each node 120. Examples of segment metadata 124 can include the following elements. · Segment ID. A unique identifier of the segment stored in the compute node 120. · HMD. Header metadata that forms part of the segment stored in the compute node 120. It may be a copy of header metadata that was originally in another segment derived from the same object. This is included in the current segment to facilitate independent processing of the current segment. · FMD. Footer metadata that forms part of the segment stored in the compute node 120. It may be a copy of footer metadata that was originally in another segment derived from the same object. This is included in the current segment to facilitate independent processing of the current segment. · Location. The location where node 120 can access the current segment. It is represented in any suitable way that node 120 uses when addressing data, such as by disk drive, logical block address (LBA), volume, file, aggregation, etc.

[0040] Similar to object metadata 112, segment metadata 124 may also be stored redundantly to enhance reliability. In some examples, node 120 may store segment metadata 124 together with the segment 170 that the metadata describes. For example, the segment metadata of segment A may be stored together with segment A. Similarly, the segment metadata of segment B may be stored together with segment B. Segment metadata 124 may be protected in the same way that segment 170 itself is protected. Various examples of segment protection are described below.

[0041] FIG. 2 shows a more detailed example of gateway 110. In this example, it is assumed that gateway 110 executes the indicated function itself. As described above, some of the functions may be executed by other computers including computing nodes 120 of cluster 130.

[0042] As shown, gateway 110 includes a type detector 210, a splitter 220, a converter 230, and a distributor 240. Type detector 210 reads a series of regions of data object 160, for example, by sampling bytes at the beginning of the object, and performs the function of identifying the object type of data object 160 based on the sampling. Type detector 210 may notify splitter 220 and converter 230 of the determined object type.

[0043] Splitter 220 performs a function of splitting data object 160 into parts 250. Each part 250 includes a respective processable unit of data object 160 and is defined by a boundary 252 within the data object. The boundary detector 222 of splitter 220 scans the boundary 252 of data object 160, i.e., the separator between processable units, and records the position of boundary 252 with respect to data object 160 (e.g., based on byte position). As described above, the nature of boundary 252 depends on the object type of data object 160, which is preferably known based on the operation of type detector 210.

[0044] In some examples, such as when splitting a Parquet file, boundary detector 222 can identify all boundaries 252 within data object 160 and define a new part 250 between each pair of boundaries. Detecting all boundaries works well in Parquet files where boundaries 252 are based on row groups and row groups tend to be large (e.g., in the megabyte range). However, if a row group is determined to be abnormally small, the boundary can be skipped and multiple row groups can be included within one part 250. In other examples, such as when splitting a CSV file, boundary detector 222 does not mark all boundaries of data object 160. This is because doing so would generate an undesirably large number of small parts 250. In such cases, boundary detector 222 may be configured not to start detecting boundaries 252 until the scanned size of the current part 250 exceeds a certain desired target size when scanning the current part 250. When the scan exceeds the target size, boundary detector 222 starts detecting boundaries and preferably identifies the first boundary that the object crosses beyond the target size. In this way, the current part ends and a new part can be started at the first detected boundary.

[0045] When the boundary detector 222 scans the object 160 for the boundary 252, the feature detector 224 may scan the object for additional features that can provide useful information relevant to subsequent processing. If the presence or absence of specific content is known in advance, it has been found that specific processing tasks can be executed faster. As a specific example, if it is known in advance that there are no quotation marks in the data, certain queries for CSV files can be executed faster. Accordingly, the feature detector 224 may check the CSV file for the presence or absence of quotation marks and update the object metadata 112 ("features") accordingly.

[0046] When a portion 250 of the data object 160 is identified based on the boundary 252, the converter 230 converts the portion 250 into each segment 170. For example, the converter 230 modifies at least some of the portions 250 by adding metadata found in some portions to one or more other portions so that such portions are more suitable for independent processing, i.e., by removing dependencies between the portions 250. The manner of adjustment depends on the object type known based on the operation of the type detector 210. The processing result of the converter 230 is the segment 170, which provides an independently processable unit of the data object. For example, each segment 170 is rendered as the same object type as the data object 160. Accordingly, the segment 170 can be processed in the same way as the data object, but the main difference is that the segment 170 is much smaller and easier to handle.

[0047] Next, the distributor 240 distributes the segments 170 to the selected nodes 120 of the storage cluster 130 and stores them in the nodes. At this time, the gateway 110 updates the object metadata 112 and records the destination of the segment 170, e.g., the identification information of a specific node 120. In this way, the data object 160 is split, converted, and distributed among the nodes 120 of the storage cluster 130.

[0048] Figures 3A and 3B show an example arrangement for splitting and converting a data object 160a containing tabular data such as a CSV file. Figure 3A shows an example of the result of splitting, and Figure 3B shows an example of the result of conversion.

[0049] As shown in Figure 3A, the data object 160a has a first row 310 and additional rows labeled from 2 to 8 (see the first column). The data object 160a has four columns. At the end of each row there is a <newline> character, which functions as a row delimiter in CSV.

[0050] When splitting the data object 160a, the splitter 220 may apply a target size 320 that defines the minimum size of a portion 350 of the data object 160a. For example, the splitter 220 can identify a position along the data object 160a corresponding to the target size 320 (illustrated by a dotted line) and split the data object 160a at the first boundary following the identified position. In the illustrated example, the splitter 220 detects the newline character at the end of the sixth row as the first boundary 252 following the target size 320 and splits the object 160a at this position. As a result, the first six rows of the object 160a form a first portion 350a, and the next two rows form the first two rows of a second portion 350b. By continuing to scan the object 160a, additional rows can be added to the second portion 350b.

[0051] Even if the splitter 220 successfully separates the object 160a at the row boundaries (thus avoiding different parts of the same row being assigned to different portions 350), the result of the splitting may still be inefficient. For example, if the first row 310 of the object 160a is a header row (e.g., a row containing text indicating column names), the second portion 350b will not have that header, and subsequent processing may be impaired. For example, the header may be required to respond to certain queries or other activities. However, this deficiency can be addressed by the converter 230.

[0052] Figure 3B shows an example of the correction result by the converter 230. Here, the parts 350a and 350b are rendered as segments 370a and 370b, respectively. The segment 370b is corrected by the insertion of the first line 310a which is a copy of the first line 310 found in the first segment 370a. By adding the first line 310a, the second part 370b is converted into a substantially independently processable unit. The changes made to the segment 370b are repeated in other segments 370 created for the object 160a, and it should be understood that all segments 370 are made to have the same first line 310 as the first segment 370a. In this way, all such segments 370 are made to be independently processable.

[0053] Note that there are CSV files that do not use a header line, and there may be cases where the first line 310 contains data instead of text-based field names. In such cases, the replication of the first line 310 of the first segment 370a of the object 160a to other segments 370 may simply propagate redundant data. However, such cases can be easily handled. For example, a query or other processing task (e.g., arriving from a client of a storage cluster) can specify whether the CSV file represented by the object 160a contains a header. If it does, the copy of the header was appropriate and need not be changed. However, if the task specifies that the CSV file does not contain a header, it will be found that the copy was unnecessary. In such cases, the node 120 that executes the distributed processing task on the CSV file can be instructed to simply ignore all first lines other than the first segment 370a of the segment 370. Little is lost as a result of copying the first line 310, and it is usually a size that can be ignored compared to the segment 370.

[0054] Figures 4A and 4B show an example arrangement for splitting and converting a data object 160b containing column-based data such as a Parquet file. Figure 4A shows an example of the Parquet file structure before splitting and conversion, and Figure 4B shows an example of the result after splitting and conversion.

[0055] As shown in Figure 4A, the Parquet file 160b starts with a 4-byte "magic number" ("PAR1") and ends with a 4-byte "magic number" ("PAR1") as described above. The file 160b further includes a plurality of row groups 410 (from 1 to N, where "N" is any positive integer) and a footer 420. The row groups 410 are typically large structures on the order of megabytes each. The footer 420 contains the metadata of the file and includes the metadata of the row groups indicating the positions (e.g., byte positions) of the row groups 410 within the file 160b. The footer 420 also includes a 4-byte data element encoding the "file metadata length".

[0056] Unlike the CSV example, if the boundary 252 can be directly detected while scanning the object forward, the boundary between the row groups 410 can be easily detected simply by reading the footer 420. That is, the splitter 220 typically passes through the entire file 160a before reaching the footer 420 and then splits retrospectively. The splitting is generally done at the boundaries of each row group such that each part 260 of the Parquet file 160b contains a single row group 410. Since the size of the row groups 410 can vary depending on the content, it is sometimes desirable to place two or more row groups 410 in a single part 260. This is a matter of design preference.

[0057] As shown in FIG. 4B, the Parquet file 160b of FIG. 4A is rendered as N different segments 470 (from 470-1 to 470-N), and each segment contains a single row group. For example, segment 470-1 contains row group 1, segment 470-2 contains row group 2, and so on up to segment 470-N, which contains row group N.

[0058] The changes shown in FIG. 4B may be implemented by the converter 230 to make each row group a self - contained Parquet file. For example, each of segments 470-1 to 470-N contains the magic number "PAR1" at the beginning and end. Also, each of segments 470-1 to 470-N may contain a modified footer, which may be a modified version of the footer 420. The footer of each segment 470 is prepared such that its row group metadata is limited to only the row group (or row groups) contained in that segment, excluding the row group metadata of row groups not contained in that segment. Further, a "file metadata length" is provided for each segment to reflect the actual length of the file metadata of each segment. Thus, each of segments 470-1 to 470-N is provided as a complete Parquet file in itself and can be processed independently, like other Parquet files.

[0059] In some examples, an additional segment 470-(N + 1) may be provided as the last segment of the Parquet file 160b. Segment 470-(N + 1) does not contain a row group but rather a persisted version of the portion of the original footer 420 of the file 160b, that is, it provides "file metadata (for all row groups)" and "file metadata length". This segment is provided for reference and may help speed up certain processing tasks, but it is not intended to be treated as a self - contained Parquet file. Nor is it intended to be used as a data source when executing queries.

[0060] Figures 5A and 5B show an example arrangement for splitting and converting a data object 160c containing video data such as a video file or stream. Figure 5A shows an example sequence of video frames before splitting and conversion, and Figure 5B shows an example result after splitting and conversion.

[0061] As shown in Figure 5A, the data object 160c includes a series of frames 510, and in the illustrated example, one or more I-frames (e.g., 510-1, 510c), one or more P-frames (e.g., 510-2, 510-3, 510a, 510d, 510e), and one or more B-frames (e.g., 510b). As is well known, an I-frame is a video frame that includes a complete image and does not depend on other frames for completeness. In contrast, P-frames and B-frames are incomplete and depend on other frames for completeness. A P-frame typically references the previous frame, while a B-frame may reference forward or backward. I-frames are larger in size than P-frames and B-frames and are costly to store and transmit, so usually, I-frames do not appear much more frequently than P-frames and B-frames.

[0062] The splitting of the video data in the object 160c operates in the same manner as the splitting of the CSV data in the object 160a (Figures 3A and 3B). For example, the splitter 220 may aim to generate a portion 250 having a size equal to or slightly larger than the target size 320. The splitter 220 attempts to find the first boundary 252 of the data object that occurs after passing through the target size. To detect the boundary of the video data, the splitter 220 may be configured to identify I-frames. Since I-frames do not need to reference previous or subsequent frames, they provide a natural boundary. In the illustrated example, the splitter 220 identifies the next boundary exceeding the target size 320 as the I-frame 510c.

[0063] However, if the video is split immediately before the I-frame 510c, since the B-frame 510b references the I-frame 510c, there will be a problem that it cannot be rendered without it. Assuming that the splitter 220 splits the video immediately after the B-frame 510b, a gap will appear in the video in the segment including the B-frame 510b. Therefore, that segment will depend on another segment and will be incomplete.

[0064] Figure 5B shows an example of a solution. Here, the object 160c processed so far is rendered as two segments 570a and 570b. To resolve the dependency, a copy 510cc of the I-frame 510c is provided to the segment 570a. The copy 510cc provides the necessary reference from the B-frame 510b and avoids the omission of video frames when rendering the segment 570a. On the other hand, the segment 570b holds the I-frame 510c as the first frame and provides an independent reference for starting the segment 570b. Subsequent frames, such as 510d and 510e, may depend on the I-frame 510c for completeness, but none of the subsequent frames reference frames before the I-frame 510c. In this way, each of the segments 570a and 570b is rendered as an individually independently processable unit without depending on other segments for completeness.

[0065] Figure 6 shows an example of an arrangement for performing distributed processing according to an additional embodiment. The illustrated arrangement may be implemented in the environment 100 of FIG. 1 or other environments. The following description assumes an implementation in the environment 100 such that the above-described features form part of an immediate embodiment. In other examples, the arrangement of FIG. 6 may be implemented in other environments having different features. Therefore, the above features should be considered illustrative and not essential unless specifically shown.

[0066] As shown in FIG. 6, the gateway 110 includes components that assist in performing distributed processing. These include, in addition to the object metadata 112 described above, a task requester 610, a dispatcher 620, an output receiver 630, and an output aggregator 640.

[0067] In a processing example, the task requester 610 initiates a request 650 to execute a processing task on a specified data object 160 (or a set of objects 160). Various types of tasks are envisioned. These may include, for example, reading and / or querying of specified data (such as tabular or tree-based data objects). Types of queries may include SQL (Simple Query Language) queries, key-value lookups, noSQL queries, etc. Tasks for video data objects may include distributed video processing tasks such as searching for specified graphic content (faces, license plates, geographical features, etc.). Tasks for audio data objects may include searching for spoken words, voice characteristics (tone, accent, pitch, etc.), specific sounds, etc. Basically, tasks that can be divided among multiple nodes 120 and potentially involve access to large amounts of data are preferred candidates for processing in the arrangement of FIG. 6.

[0068] When the request 650 is issued, the dispatcher 620 begins distributing the components of the requested task to each node 120. For example, the dispatcher 620 checks the object metadata 112 to identify the segments 170 of the specified data object 160 (or set of objects) and their respective locations within the storage cluster 130. In the illustrated simplified example, the object metadata 112 identifies three segments 170 (e.g., S1, S2, and S3) that make up the data object 160 (typical results may include dozens or hundreds of segments) and three compute nodes 120-1, 120-2, and 120-3 that store each segment 170.

[0069] The dispatcher 620 then sends requests 650-1, 650-2, and 650-3 to the identified nodes 120-1, 120-2, and 120-3, respectively. Requests 650-1, 650-2, and 650-3 may be similar or identical to request 650. For example, the same query or other task specified in request 650 may be provided. However, such requests 650-1, 650-2, and 650-3 need not be identical to each other. For example, some of the requests may include segment-specific metadata (e.g., stored in object metadata 112) that is different from what was sent in other requests and may be used to direct processing tasks at specific nodes.

[0070] The identified nodes 120-1, 120-2, and 120-3 each receive requests 650-1, 650-2, and 650-3, and each of these nodes begins to execute the requested tasks on their respective segments. For example, node 120-1 executes the task for segment S1, node 120-2 executes the task for segment S2, and node 120-3 executes the task for segment S3. In one example, each node 120 need not contact other nodes 120 and independently executes each task on each segment 170. For example, node 120-1 completes its work by accessing only S1 and does not need access to S2 or S3. The same applies to other nodes.

[0071] When nodes 120-1, 120-2, and 120-3 execute each task, such nodes generate respective outputs 660 shown as output 660-1 from node 120-1, output 660-2 from node 120-2, and output 660-3 from node 120-3. The participating nodes each send their respective outputs 660 back to the gateway 110, and the gateway 110 collects the outputs at the output receiver 630.

[0072] As shown in the enlarged view near the bottom of FIG. 6, output receiver 630 can receive outputs 660 from participating nodes 120 in any order. In a first scenario, nodes 120-1, 120-2, 120-3 are configured to wait for their respective tasks to complete before sending back the output. In this case, the outputs 660 from a particular node may arrive all at once, or the outputs from different nodes may arrive at different times based on their respective completion times. Output data 662 shows an example of the result according to this first scenario. Here, the output 660-2 from node 120-2 arrives first and thus appears first in the output data 662, followed by the arrival of output 660-1 (from node 120-1), and then the last arriving output 660-3 (from node 120-3). The outputs 660 are thus interleaved in the output data 662.

[0073] In a second scenario, nodes 120-1, 120-2, 120-3 are configured to return their outputs incrementally, such as immediately when an increment becomes available. In this second scenario, each participating node can return its output 660 in multiple transmissions, which may be spread out over time. Output data 664 shows an example of the result based on this scenario. Here, it can be seen that the output data 664 includes six different batches (660-1a, 660-1b, 660-2a, 660-2b, 660-3a and 660-3b). That is, two batches of the output from each of nodes 120-1, 120-2 and 120-3. Since the batches appear in the output data 664 in the order of reception, they may be interleaved with a finer granularity than seen in the first scenario.

[0074] Of course, the gateway 110 may sort the output 660 in any way, and any node 120 of the storage cluster 130 may be called to perform this task. In some examples, both the affected node and the gateway 110 may participate in sorting the output 660. For example, each node may sort its respective output so that the results 660-1, 660-2, or 660-3 arrive individually in the sorted order. The gateway 110 may complete the task, for example, by using the aggregator 640 to sort the sorted set of returned results.

[0075] Sorting takes time, and in many processing tasks, speed is more important than sorted output. To further promote high-speed operation, in some examples, the compute node 120 may employ RDMA (Remote Direct Memory Access) when returning the output 660 to the gateway 110.

[0076] In some processing tasks, the dispatcher 620 may send the processing request to all participating nodes (i.e., all nodes storing segments of the target data object). In other examples, the dispatcher 620 may limit the nodes to which the request is sent, for example, based on knowledge of the content of the preceding segment, the byte range of the segment, or other factors. By limiting the number of participating nodes in this way, traffic on the network 140 (FIG. 1) can be reduced and efficiency can be further enhanced.

[0077] Some processing tasks may involve aggregation. For example, a query may request the number of records that meet specified conditions rather than the records themselves. The query may request an average value, a maximum value, a minimum value, or other aggregated values. Node 120 may perform certain aggregation functions (such as count, sum, maximum, minimum, etc.) on its own, but individual nodes 120 typically do not aggregate outputs across multiple nodes. Rather, this function can be performed by data aggregator 640. For example, aggregator 640 can receive counts from multiple nodes, and each node provides a partial aggregation result derived from the processing of each segment. Aggregator 640 can sum the counts received from the responding nodes to generate an aggregated total for the entire data object 160. For example, to generate an aggregated average of data objects, aggregator 640 can instruct each participating node to provide both a count and a sum. Next, all the returned counts are summed to generate an aggregated count, all the sums are summed to generate an aggregated sum, and the aggregated sum is divided by the aggregated count to generate the desired aggregated average. Other types of aggregation functions can be performed in a similar manner.

[0078] It should be understood that the arrangement of FIG. 6 can execute aggregation queries at very low cost from the perspective of bandwidth. Since each participating node calculates a local aggregation and returns only the result, the aggregation query can be executed across a very large dataset, and the output 660 is very small, usually less than 1 kB and often on the order of a few bytes.

[0079] Gateway 110 has been shown and described as the originator of request 650, as the dispatcher of requests to the affected nodes, and as the collector of output 660 from the nodes, but these functions may alternatively be performed by other computers or by multiple computers. In fact, these can be performed by one or more nodes 120 of storage cluster 130. Therefore, the example shown is intended to be illustrative rather than limiting.

[0080] Figures 7 and 8 illustrate an exemplary arrangement for performing data protection of segment 170 according to additional embodiments. The arrangements shown in FIGS. 6 and 7 may be implemented in the environment 100 of FIG. 1 and / or FIG. 6, or in an environment different from the environments illustrated above.

[0081] FIG. 7 shows a plurality of segments 170 generated from a single data object 160, with the segments 170 arranged vertically. Although not required, the segments 170 can be arranged in order, in which case the earliest generated segment (closest to the beginning of the object) is displayed at the top, and vertically adjacent segments 170 correspond to adjacent portions of the data object 160. Nine segments 170 are shown, but more than nine segments 170 may be generated from the data object 160. In one example, the nine segments 170 depicted are the first nine segments generated from the data object (e.g., by splitter 220 and converter 230; FIG. 2).

[0082] Notably, each of the segments 170 has a different length. Thus, as shown in the upper right of the figure, it is possible to rank the segments 170 in order of length, e.g., from the longest to the shortest.

[0083] FIG. 8 is an enlarged view of the similarly ranked segments 170. Here, a K+M erasure coding process is performed on nine segments (K = 9) (e.g., by gateway 110), generating M = 3 elements 810 of repair data that provide various forms of parity information. The K segments, together with the M repair elements, constitute a repair group 802 that includes a total of 12 elements in all.

[0084] The illustrated repair group 802 tolerates damage to up to M elements before data loss occurs. The damaged elements can be any elements of the repair group 802 that include data segments 170 and / or repair elements 810 in any combination. As long as not all M elements are damaged, complete recovery and repair are possible. It should be understood that the choices of K = 9 and M = 3 can be changed based on the desired level of data protection and other factors. In one example, a computationally efficient procedure 800 that seems entirely new is used to generate the repair elements 810.

[0085] In conventional erasure coding schemes, the lengths of all K data elements may have to be equal. If the lengths of the data elements are different, zero-padding may be used to make the lengths equal. Next, parity calculations are performed using the total length of all K data elements, and M parity data elements having the same length as the K data elements are generated.

[0086] In contrast to conventional erasure coding techniques, procedure 800 generates repair data elements from data elements of different lengths. Zero-padding is not required. In one example, procedure 800 proceeds by logically aligning segment 170, i.e., the K = 9 data elements. For example, segment 170 may be aligned at each respective top as shown. Alternatively, segment 170 may be aligned at each respective bottom (not shown), or may be aligned in some other known way. Note that such alignment is logical, not physical. No actual movement of segment 170 is necessary. Also, the illustrated ranking of segment 170 is to be understood as logical, not physical.

[0087] With segment 270 logically aligned, procedure 800 proceeds by identifying the shortest segment 170 (labeled "1") and identifying the corresponding range (Rng1). Rng1 is aligned with segment 1 and has the same size and limits. Since segment 1 is the shortest segment and segment 170 is logically aligned, there is data within Rng1 for all K segments 170 (segments 1 - 9). Using the Rng1 data across segments 1 - 9, the procedure calculates M repair data sets, places one set each at the M repair elements 810, and places the repair data at the Rng1 positions of the respective repair elements 810. The repair data for Rng1 is thus completed, and such repair data is based on all K segments 170. It should be understood that the calculation of the repair data described here may be similar to that used in conventional K+M erasure coding. The details are not important for the embodiments and will not be described further.

[0088] Thereafter, procedure 800 continues for additional ranges in a similar manner. For example, Rng2 corresponds to the portion of segment 2 that extends beyond segment 1, i.e., the portion of segment 2 for which repair data has not yet been calculated. Since there is no Rng2 data in segment 1, the repair data for Rng2 can be calculated using only the corresponding portions of segments 2 - 9 (i.e., a total of K - 1 segments). Similar to the previous procedure, M repair data sets are calculated, placed one set each at the M repair elements 810, and this time the repair data is placed at the Rng2 positions. The repair data for Rng2 is thus completed, and such repair data is based on only K - 1 segments 170.

[0089] Procedure 800 can continue in this way for each range from Rng3 to Rng8. In the calculation of the repair data for each range, one less segment is involved than in the calculation of the previous range. Thus, K - 2 segments are involved in the calculation of Rng3, K - 3 segments are involved in the calculation of Rng4, and similarly, K - 7 segments, i.e., only segments 8 and 9, are involved in the calculation of Rng8. Since Rng9 intersects with only a single segment (segment 9), no calculation is required for Rng9. Instead of calculating the repair data for Rng9, in procedure 800, the affected data, i.e., a replica (copy) of the portion of segment 9 within Rng9, is saved. Individual copies of the Rng9 data can be provided at the Rng9 positions of each repair element 810.

[0090] The erasure coding procedure 800 can typically be calculated faster than conventional erasure coding. Instead of requiring all K data elements to calculate the repair data for M repair elements 810, in procedure 800, only K data elements are required for the shortest data elements. For each successive shortest data element, one less data element is required in procedure 800, and ultimately only two data elements are required, thereby reducing the computational complexity and execution time.

[0091] The segments 170 generated from object 160 can be protected using the erasure coding procedure 800. For example, when distributing the segments 170 to the compute nodes 120 for storage in the cluster 130, the gateway 110 (or other computer) can execute the procedure 800 with reduced computational cost to generate the repair elements 810. The procedure 800 operates on K segments 170 at a time, generating M repair elements for each, and forming respective repair groups 802 for each set of K + M elements.

[0092] FIG. 9 shows an example of the arrangement of a plurality of repair groups 802 that can be used to protect a specific data object 160x. As shown, groups up to repair groups 802-1, 802-2, and 802-R provide data protection for the data object 160x, for example, using an erasure coding procedure 802. The first repair group 802-1 includes and protects a group of the first K segments 170 generated from the data object 160x, the second repair group 802-2 includes and protects a group of the second K segments 170 generated from the same data object 160x, and so on, continuing up to the Rth repair group 802-R that protects the group of the last segments 170. Note that the repair group 802-R includes less than K segments. For example, the data object 160x may have ended (run out of data) after generating only 7 segments. The segments 170 that make up the repair groups 802 are arranged in columns (column 1 to column 9), and each column corresponds to each of the K elements.

[0093] It should be understood that erasure coding may impose certain constraints on data placement. For example, two segments 170 belonging to the same repair group 802 should not normally be stored on the same disk drive (e.g., SSD, magnetic disk drive, etc.). This is because if so, the redundancy of erasure coding will be compromised and the segments will be exposed to an increased risk of data loss. For similar reasons, two segments 170 belonging to the same repair group 802 should not normally be stored on the same compute node 120. This is because if so, for example, the redundancy will decrease in the event of a failure of the compute node 120. However, these rules are not usually applied to the entire different repair groups 802. For example, storing segments 170 belonging to different repair groups 802 on the same compute node 120 will not result in a substantial loss of redundancy, provided that the two segments do not belong to the same repair group 802. For example, it may be acceptable for a single compute node 120 to store one segment 170 (a total of R segments of the same data object) from each of the R repair groups that protect a given data object 160.

[0094] Furthermore, it should be understood that erasure coding is one way to protect data, and another way is replication. As an example, data objects 160 and their associated repair data and / or replicas exist in buckets of an object store, and the data protection scheme is applied per bucket. Buckets that use replication for data protection use replication to protect all of its contents, including all the objects contained therein. Similarly, buckets that use erasure coding for data protection use erasure coding for all of its contents. The erasure coding parameters K and M can also be selected and applied per bucket. Thus, in the arrangement of FIG. 9, since the bucket containing object 160x is using these settings, erasure coding can be used with K = 9 and M = 3, which is then applied globally to all the contents within the bucket.

[0095] Figure 10 shows an example of a method 1000 for determining various quantities used in the management of data object 160 and its segments 170. This method 1000 is premised on data protection using erasure coding and can be used to determine the desired target size 320 (Figure 3) of segment 170, and can also be used to determine the number R (Figure 9) of repair groups 802 used to protect data object 160. Method 1000 can be executed, for example, by gateway 110, node 120 of storage cluster 130, or other computers connectable to cluster 130. At the start of method 1000, it is assumed that the size of data object 160 and the value K (used in K+M erasure coding) are known in advance.

[0096] At 1010, method 1000 determines the maximum size S of segment 170 that can be efficiently processed by node 120 MAX . The maximum size can be determined based on practical considerations such as the hardware specifications of node 120 (e.g., clock speed, number of cores, amount of memory, etc.), and the expected latency to processing tasks and user expectations. For example, S MAX can typically range between several hundred kilobytes and several megabytes.

[0097] At 1012, the method calculates the average number of bytes B C per column. In one example, the value of B C may be based on the size of data object 160, "object size", and the value K used in the K+M erasure coding used to protect data object 160. For example, B C = object size / K. Briefly referring back to Figure 9, it can be seen that B C represents the average amount of data per column in the illustrated columns.

[0098] At 1014, method 1000 divides, for example, B C by S MAXDivide by and round up to the nearest integer to calculate the number R of repair groups. More specifically, the number of repair groups is R = B C / S MAX and is calculated and rounded up.

[0099] In 1016, this method calculates the target segment size 320 as S TAR = B C / R. The resulting quantity S TAR may be provided to the splitter 220 when determining, for example, the position to start searching for the boundary 252 when dividing the data object 160.

[0100] In 1018, the method 1000 instructs the splitter 220 to divide the data object 160 in a way that generates a portion 250 that extends to at least the next boundary 252 of S TAR such as generating a portion 250 of the same size as S TAR or more.

[0101] The method 1000 provides useful guidelines for determining the target segment size 320 and the number R of repair groups to be used for a particular data object 160. The actual selection of these quantities may be at the discretion of the administrator and may also be determined by factors other than those described. Therefore, the method 1000 is intended to be advisory rather than mandatory.

[0102] FIG. 11 shows an exemplary computing node 120 in more detail. The computing node 120 is intended to represent the computing nodes 120-1, 120-2, and 120-3 of the storage cluster 130. It is also intended to represent the gateway 110 of FIG. 1.

[0103] As shown, computing node 120 includes one or more communication interfaces, such as one or more network interface cards (NICs) 1110, a set of processors 1120, such as one or more processing chips and / or aggregates, memory 1130, such as volatile memory for executing software, and persistent storage 1140, such as one or more solid state disks (SSDs) or magnetic disk drives. The set of processors 1120 and the memory 1130 together form a control circuit and are configured and arranged to perform various methods and functions as described herein. Also, the memory 1130 includes various software configurations implemented in the form of executable instructions, as shown in FIGS. 1 and 2. When the executable instructions are executed by the set of processors 1120, the set of processors 1120 performs the operations of the software configuration. In one example, one or more of the set of processors 1120 reside on the network card 1110, facilitating high-speed communication over the network 140 and potentially improving bandwidth and efficiency.

[0104] FIGS. 12, 13, and 14 show exemplary methods 1200, 1300, and 1400 that may be implemented in relation to environment 100 and provide an overview of some of the features described above. Methods 1200, 1300, and 1400. Such methods are typically executed by software configurations, for example, as described in relation to FIGS. 1 and 2. The various operations of methods 1200, 1300, and 1400 may be ordered in any suitable manner. Accordingly, embodiments may be constructed in which the operations are performed in an order different from the illustrated order, which may include performing some operations simultaneously.

[0105] FIG. 12 illustrates an exemplary method 1200 for managing data objects. At 1210, a data object 160 is divided into a plurality of portions 250 at a boundary 252 within the data object 160 (see FIG. 2). The boundary 252 provides a separator between processable units 250 of the data object 160 according to the type of the data object (e.g., CSV, JSON, XML, Parquet, video, etc.). At 1220, the portions 250 are converted into segments 170 that provide independently processable units of the same type as the data object 160. For example, data and / or metadata may be copied from one portion 250 to another, and other changes may be made to reduce or eliminate dependencies between segments 170. At 1230, the segments 170 are distributed to and stored in a plurality of compute nodes 120 of a storage cluster 130.

[0106] FIG. 13 illustrates an exemplary method 1300 for managing data objects. At 1310, a data object 160 is divided into a plurality of segments 170, for example, by the operation of a splitter 220 (FIG. 2). At 1320, the segments 170 are distributed to a plurality of compute nodes 120 of a storage cluster 130. At 1330, a distributed processing task is executed by the storage cluster 130. The distributed processing task is executed independently by each of the plurality of compute nodes 120 of the storage cluster 130 on each segment 170 or set of segments 170 stored therein.

[0107] FIG. 14 illustrates an exemplary method 1400 for managing data objects. At 1410, a data object 160 is divided into a plurality of segments 170, and at least some of the segments 170 have different lengths from each other (see FIGS. 7 and 8). At 1420, the segments 170 are distributed among a plurality of compute nodes 120 of a storage cluster 130. At 1430, K of the segments 170 are protected using M elements 810 of repair data generated from the K segments, and each of the M elements 810 stores repair data calculated from each grouping of segments selected from the K segments (e.g., one grouping having K segments, one grouping having K - 1 segments, etc.) in a plurality of ranges (e.g., Rng1, Rng2, etc.).

[0108] An improved technique for managing a data object 160 in a storage cluster 130 includes dividing the data object 160 into a plurality of portions 250 at a boundary 252 within the data object 160. The technique further includes converting the portions 250 of the data object 160 into segments 170 that provide independently processable units, and distributing and storing the segments 170 among a plurality of compute nodes 120 of the storage cluster 130.

[0109] Section II: Partitioning, Processing, and Protecting Multidimensional Data In this section, an example of managing multidimensional data will be described. It should be understood that any of the features and methodologies as described in Section I above can also be used in the embodiments described in this Section II. However, the specific embodiments of Section II may be used independently of those described in Section I. Thus, unless otherwise stated, the features of Section I should not be considered necessary for any of the features of Section II described below.

[0110] The vast amount of multi-dimensional array-oriented data generated in the scientific community is mainly stored in the industry-standard Network Common Data Form (NetCDF). An important issue in using data stored in NetCDF is that the dataset is often too large for all users to copy or transfer via the network, and every time an analysis tool accesses the data, it is necessary to retrieve the data, extract subsets, and format the extracted subsets. These operations can sometimes account for 80 - 90% of the total time required for insights.

[0111] To unleash the vast potential of terabyte-scale NetCDF-formatted data stored in different locations, we are developing a solution that integrates in-situ analysis capabilities for multi-dimensional data (such as NetCDF) into our highly innovative real-time smart data lake solution. Dramatically accelerating data analysis executed in the storage layer addresses the major issues of reducing data traffic between sites, reducing computing resources, and reducing costs. At the same time, significantly accelerating data analysis has the potential to bring great benefits to the scientific community.

[0112] In this regard, new techniques for managing multi-dimensional data include providing a raw dataset that includes data arranged along multiple dimensions such that each dimension covers its respective original range in dimension units. This approach further includes extracting a plurality of data portions from the raw dataset, each portion covering a reduced range of dimension units that is smaller than the original range in at least one dimension, and all of the extracted portions together covering the original range of the raw dataset in all dimensions.

[0113] FIG. 15 shows an exemplary arrangement for dividing a multi-dimensional dataset 1502 into segments 170 that can be processed independently, which segments can be stored in nodes 120 of a storage cluster 130 (FIG. 1). Such nodes 120 may be configured to perform in-situ data analysis in a highly parallel and distributed manner at the storage level.

[0114] As shown, the exemplary dataset 1502 has three dimensions labeled X, Y, and time, thereby providing an N-dimensional space 1510, where N is equal to 3 (time is considered one dimension of the N-dimensional space for this purpose). The N-dimensional space 1510 can include any number of dimensions. In one example, the dataset 1502 is provided as a NetCDF file, such as a NetCDF4 (or later) file. The file format of the NetCDF4 file may be, for example, HDF (Hierarchical Data Format) subsequent to HDF5. However, these are merely examples. A header 1504 may be included in the dataset 1502 to define, for example, dimensions, variables, and other features.

[0115] The dataset 1502 can be divided along any dimension. In one example, the division is performed by a splitter 220 described in connection with FIG. 2 above. The three-dimensional space 1510 may be rendered as a plurality of parts 250, each part 250 having a one-dimensional size in one dimension (e.g., time), but maintaining the original dimensional size in other dimensions (e.g., X and Y). Viewed as a whole, the parts 250 cover the same range of the N-dimensional space as the original dataset 1502, but each part 250 is dimensionally smaller than the space 1510. Generally, it is most optimal to perform the division along the least varying (innermost) dimension. In this case, that is time. However, other methods are also conceivable.

[0116] In one example, splitter 220 is configured to split dataset 1502 to create portions 250 (FIG. 2) of a desired target size, such as 4MB, 8MB, etc. The desired size can vary based on many factors, such as the size of the dataset, the number of available nodes 120, the computing power and memory of each node, etc., and the example shown is not limiting. For some datasets and desired sizes, splitting into a single unit may not be necessary. For example, each portion 250 may include units of two or more times while approximately conforming to the desired size. For other datasets or datasets of desired sizes, splitting into a single unit may still result in a file size that is too large and thus may not be sufficient. In such cases, one or more additional splitting iterations may be performed in one or more other dimensions, such as Y and / or X. For example, after splitting time into a single unit, Y (the dimension with less variation next) may also be split and can be split into multiple Y units or a single Y unit. If splitting both time and Y into single units still results in a size that is too large, splitting may proceed up to X, with the limit being to split time, Y, and X into single units respectively. In most datasets, generally, one-dimensional or two-dimensional splitting is sufficient.

[0117] Once the desired level of splitting is achieved and portions 250 are identified, converter 230, which may be executing within gateway 110 (FIG. 2) for example, can convert portions 250 into corresponding segments 170. In one example, converting portions 250 into segments 170 includes rendering each portion 250 as an independently processable unit, such as a NetCDF4 file in HDF5 format. The conversion may include constructing a header 1530 for each portion and providing index data and other metadata necessary to support the HDF5 format.

[0118] As an example, both the above-mentioned splitting and conversion can be easily performed by using the NetCDF4 and HDF5 libraries. For example, the HDF5 library can be used to extract the information necessary to construct segment 170. Also, the netcdf-c library can be used to construct a header 1530 suitable for each segment. In one example, the data of each segment 170 may be placed in a determined location and adjusted with respect to the data index and header information.

[0119] The header 1530 of each segment 170 reflects the dimensional range of each part 250, rather than the dimensional range of the original data set 1502. Using the simple example shown in FIG. 15, the header 1504 of the original data set (e.g., NetCDF4 file) 1502 has dimensions such as: Dimensions: Time = 5; Y = 3; X = 8; may have. In contrast, the header 1530 of each segment 170 has dimensions such as: Dimensions: Time = 1; Y = 3; X = 8; may have.

[0120] When segment 170 is formed from each part 250, the distributor 240 can place the segment 170 on each node 120 (such as node 0 to node 4, etc.) of the storage cluster 130 as shown in the figure. In one example, each of these nodes is made to include segment 170 including a NetCDF4-compliant dataset, such as a file that can be accessed using the NetCDF4 or HDF5 library. Access to segment 170 using these libraries enables on-site access to the data and independent data analysis. For example, the part 250 shown in FIG. 15 provides a time slice that can be analyzed individually, for example using an AI model, to provide useful spatial (X-Y) information for a specific time represented by the slice. Similar analysis is possible when dividing in other dimensions or other ways. In some examples, additional data may be added to the NetCDF4 file within the segment, for example to provide context to facilitate data processing.

[0121] In some examples, the header 1504 of the original dataset 1502 and other metadata itself can be stored in the cluster 130, for example, together with one or more parts 250, or in a separate segment 170 or other location. Also, it should be understood that placing segment 170 on each node 120 may include protecting the segment using erasure coding, as described in connection with FIGS. 7-9 above.

[0122] In the example shown in FIG. 15, a fair degree of flexibility is assumed regarding how the data is split into portions 250. For example, the data in dataset 1502 may be laid out continuously in an array-based data format such as that used in the C programming language. In this data layout, an N-dimensional array can be easily split to any extent along any dimension (or multiple dimensions). To conform to NetCDF4, any dataset must contain a complete "subtensor", i.e., a complete array in its dimensional space. A subtensor is so named because it is part of an N-dimensional space (tensor).

[0123] FIG. 16 shows a different arrangement, in which the data of dataset 1502a is logically arranged in chunks 1610. Here, each chunk 1610 represents an N-dimensional subspace of an N-dimensional space such as space 1510 in FIG. 15. For simplicity, the N-dimensional space in FIG. 16 is shown as two-dimensional only. Chunks 1610 typically store data in a compressed format, but this is not essential.

[0124] All chunks 1610 within dataset 1502a have a uniform dimensional ratio. For example, all chunks 1610 have the same dimensions such as 1×1, 1×2, 200×50, etc. However, the chunks 1610 can have non-uniform data sizes. For example, one chunk may be 4MB while another may be 5KB or less, which reflects, for example, different levels of compression and / or filtering. Thus, the goal of creating a portion 250 with a desired target size is not simply a matter of splitting an array.

[0125] As an example, the splitter 220 processes the chunked data by selectively combining the chunks 1610 to generate a portion 250 having a desired target size. For example, the splitter 220 can assign chunks 1610 that are close to the target size to each portion 250, but the splitter 220 may also combine smaller chunks into a single portion, in which case the sum of the sizes of the smaller chunks is approximately the desired target size.

[0126] In one example, the chunks 1610 may simply be combined without regard to size. As described above, in NetCDF4, a dataset is required to be provided as a subtensor. This requirement is met only when only groups of chunks are arranged in a subtensor. In the example of FIG. 16, any subtensor needs to have the same number of chunks in each row and each column (more generally, each dimension). In the two-dimensional context of FIG. 16, the chunk grouping must form a complete rectangle without holes or protrusions.

[0127] For example, the groups G1, G2, G3, and G4 are all regular chunk groupings because their constituent chunks form a complete subtensor (a rectangle without holes or protrusions). In contrast, the group G5 is not a regular grouping because there are chunks located outside the rectangle formed by the other chunks. More precisely, the group G5 is not a subtensor.

[0128] Taking this constraint into account, the splitter 220 is configured to construct a grouping of chunks that forms a subtensor with a size approximately equal to the target size. Such a grouping is then provided to each portion 250 and, as described above, can be converted to each segment 170 and stored in the cluster 130.

[0129] In some examples, splitter 220 applies additional constraints when grouping chunks 1610 into parts 250. For example, splitter 220 may further impose a requirement that all chunks joined within a part must be physically contiguous within the original NetCDF4 file.

[0130] FIG. 17 shows an exemplary physical layout 1700 of a NetCDF4 file. As shown, file 1700 includes a header 1710, metadata 1720, and variable data 1730 such as data for variables V1, V2, and V3. Generally, NetCDF4 arranges data in a logical order, such as in an adjacent array-based order as described above. However, it is not guaranteed that chunk-based data is arranged adjacently, and in some cases, the data may be scattered. For example, most of the data for variable V1 is arranged adjacently, or at least continuously, as shown, but part 1740 is arranged separately from the rest. "Adjacent" data is directly adjacent, while "continuous" data may include metadata or other data other than other chunks in between. All adjacent data is continuous, but not all continuous data is adjacent. NetCDF4 uses metadata 1720 to handle the scattered data mapping, but physically non-contiguous data can pose challenges for reconstruction and certain data processing. Thus, splitter 220 may operate such that any grouping of chunks 1610 is composed of physically contiguous chunks within the original NetCDF4 file. It should be understood that providing physically contiguous data is an optimization in certain embodiments and not essential.

[0131] In some examples, in the selection of the physically contiguous chunks described above, if the chunks are merely separated on the file layout by metadata or other data other than the chunks, it does not prevent the chunks 1610 from being grouped together. As shown, metadata 1720a is located between two consecutive (but non - adjacent) regions 1750a and 1750b of the chunk data of variable V2. Even with such metadata 1720a present, it does not prevent splitter 220 from grouping regions 1750a and 1750b together. In fact, the grouping can proceed beyond 1750b and can continue until a complete grouping (forming a sub - tensor) is created.

[0132] FIG. 18 shows an example of object metadata 112 that may be provided to facilitate the management of multi - dimensional data storage and / or processing. Object metadata 112 may include many of the same metadata elements as described above in connection with FIG. 1, but may also include additional elements useful for this data type.

[0133] For example, object metadata 112 may associate a segment 170 (segment ID) with each position within a cluster 130 where those segments are found, such as a particular node 120 and, in some cases, a position within the node such as a path name. In some examples, object metadata 112 may associate a segment with each dimensional range of the multi-dimensional data stored within the segment, such as a range of X values, a range of Y values, and a range of time values (e.g., the example of FIG. 15). In some examples, the reported dimensional ranges for a segment may be overly inclusive, and object metadata 112 may represent a range that is larger than the actual dimensional range covered by the segment. Object metadata 112 may further store various characteristics of the multi-dimensional data stored in the segment, such as characteristics that explain the nature of the stored data and / or the specificity of the data, where that knowledge can facilitate efficient reconstruction, access, or data processing. For example, the dimensions of NetCDF4 may be associated with each label that may be relevant to only a particular segment. In such cases, the characteristics stored in object metadata 112 may include the association between the dimension and the label. In a further example, object metadata 112 can store, for example, all or part of the NetCDF4 header metadata of the files included in segment 170. In some cases, such header metadata may be stored verbatim. Storing such metadata can facilitate querying and other data processing functions.

[0134] FIG. 19 shows an exemplary arrangement for distributed processing of multi-dimensional data, providing a more specific example of the arrangement shown and described above with respect to FIG. 6. The arrangement of FIG. 19 is applicable to various data processing scenarios, such as reading data, reconstructing the original NetCDF4 file, selection queries (including aggregation queries), and various types of data analysis.

[0135] As before, task requester 610 initiates request 650 to execute a processing task on a specified data object 160, which in this case can be a multi-dimensional dataset 1502 or 1502a (hereinafter referred to using a single reference numeral 1502), such as a NetCDF4 file. When request 650 is issued, dispatcher 620 begins distributing the components of the requested task to each node 120. In this example, five nodes 120, namely nodes 0-4, are identified. Next, dispatcher 620 sends requests 1910-1 through 1910-4 to each of nodes 0-4. Requests 1910-1 through 1910-4 may include, for example, read requests, selection queries, aggregation queries, or analytical processing requests. In some examples, requests 1910-1 through 1910-4 may all be the same, for example the same as request 650. In such a case, each node 120 can respond based on the content it stores. In other examples, requests 1910-1 through 1910-4 are individually tailored to the particular segments being accessed, such as by restricting the query ranges for different segments based on the ranges included in each segment read from object metadata 112.

[0136] The specified nodes 0-4 receive respective requests 1910-1 through 1910-4, and each of these nodes begins execution of the tasks requested on each segment 170. For example, node 0 executes the tasks for segment S0, and node 1 executes the tasks for segment S1. Execution of the tasks may be facilitated by local agent 1902. For example, agent 1902 can access the local segment using the NetCDF4 and / or HDF5 libraries and respond to the request by extracting data that satisfies the parameters of the request. Local agent 1902 can also calculate values such as count values, average values, total values, maximum values, minimum values, etc. based on local data and perform local aggregations. Such local processing eliminates the need to transmit large data selections over the network. In one example, each node 120 independently executes its respective task on each segment 170 without the need to contact other nodes 120. For example, node 0 does not need access to S1, S2, S3, or S4 and completes its work by accessing only S0.

[0137] When node 120 executes each task, such a node generates each output 1920, shown as outputs 1920-0 through 1920-4. The participating nodes send each output 1920 back to gateway 110, and gateway 110 collects the outputs at receiver 630. As described above, node 120 may perform some kind of aggregation function itself, but it is not typical for individual nodes 120 to aggregate outputs that span multiple nodes. This function may instead be performed by data aggregator 640. For example, aggregator 640 may receive partial aggregation results from multiple nodes. Aggregator 640 may combine the partial aggregation results to generate an overall aggregation value that can represent the entire NetCDF4 file.

[0138] Among the data processing tasks, there are those that include creating objects such as NetCDF4 files. For this type of task, the gateway 110 may include a formatter 1930. For example, the formatter 1930 is configured to set the received data (e.g., from the response 1920) to a NetCDF4 compliant file. Such processing by the formatter 1930 facilitates the reconstruction of the original NetCDF4 file (dataset 1502) and also facilitates the construction of other datasets, such as a combination of outputs from multiple segments.

[0139] Figures 20 - 22 illustrate exemplary methods that may be executed in the context of FIGS. 1 and 19. Such methods may be executed by one or more processors of the gateway 110, for example, based on instructions and data stored in the memory of the gateway 110. The various operations of such methods may be ordered in any suitable manner. Thus, embodiments may be constructed in which the operations are executed in an order different from the illustrated order, which may include executing some operations simultaneously.

[0140] Figure 20 illustrates an exemplary method 2000 for performing data reading or querying in the context of FIG. 19. At 2010, a request 650, such as a query, is received. The query may be presented in any form. In one particular example, the query is provided as an SQL (Structured Query Language) query, which may be structured, for example, as an SQL SELECT query. The request 650 may specify a set of query conditions. In the case of reading the entire dataset, the condition may be expressed as "SELECT *".

[0141] In some examples, the query condition is directly expressed as a predicate in units that match the dimensions of the dataset 1502, for example, a NetCDF4 file. In other examples, the query condition is expressed in dimensionless units, such as the units of a particular variable stored in the NetCDF4 file. In these latter examples, method 2000 may include step 2020 that provides a first stage of query processing. Here, gateway 110 contacts node 120 (e.g., all nodes) that requests a dimensional predicate corresponding to the query condition. The node returns predicate information. Next, gateway 110 converts the query condition from request 650 into a corresponding predicate expressed in terms of the dimensions defined in the NetCDF4 file.

[0142] Regardless of whether the first stage of the query is required, the process proceeds to 2030 where gateway 110 accesses object metadata 112 to obtain the location of relevant segments, such as segments that may contain the requested portion of the data. The identified segment 170 may be the segment itself that stores the requested data, or may be overly inclusive, such that more segments are identified than actually contain the requested portion of the data. Step 2030 may be optional as it may simply involve broadcasting request 650 to all nodes within cluster 130 that includes a portion of the NetCDF4 file.

[0143] In 2040, request 650 or a modified version thereof (e.g., requests 1910-0 through 1910-4) is sent to node 120 that includes the relevant segment, which may include sending request 650 to all nodes within cluster 130. This step 2040 can provide a second stage of query processing (if the first stage of processing was done at 2020) or may be the only stage of the query (if the first stage of the query was not required).

[0144] In 2050, the gateway 110 receives partial query results from each node, and in 2060, the gateway 110 merges the partial query results to generate an overall result representing the NetCDF4 file as a whole.

[0145] Figure 21 shows an example of a method 2100 for reconstructing a NetCDF4 file from segments 170 stored in cluster 130. In 2110, data is extracted from segment 170 that contains any part of the original NetCDF4 file. For example, the gateway 110 sends one or more requests 650 or 1910-X to node 120, requesting that all data for each segment be returned. In one example, the data is requested using a SELECT * query, which can be implemented as described in method 2000 above.

[0146] In 2120, the data extracted from the segment in 2110 is put into a template data set 2122 such as a template NetCDF4 file. In 2130, the gateway 110 retrieves the original header 1504 from cluster 130, for example, from a dedicated metadata segment, from object metadata 112, or from other locations within cluster 130. In 2130, the gateway 110 copies the retrieved original header 1504 to the template data set 2122. By assembling the data and header of the original NetCDF4 file in this way, the original NetCDF4 file is completely reconstructed. Preferably, the NetCDF4 file is reconstructed to be identical to the original file in every respect (assuming no updates have been made since the initial split).

[0147] Figure 22 shows an exemplary method 2200 that may be implemented in a particular embodiment and provides an overview of some of the features described above. At 2210, an original dataset 1502 is provided. The dataset includes data arranged along a plurality of dimensions of an N-dimensional space 1510. Each dimension of the N-dimensional space 1510 has its own original range in units of the dimension (e.g., the number of X values, the number of Y values, the number of time values). The original dataset has a data format such as NetCDF4.

[0148] At 2220, a plurality of portions 250 of the data are extracted from the original dataset 1502. Each portion 250 extends over a reduced range of units of the dimension in at least one dimension of the N-dimensional space (one or more time units, one or more Y units, etc.), and the extracted portions 250 together cover all of the original ranges of the N-dimensional space 1510.

[0149] At 2230, the extracted portions 250 are rendered into respective segments 170 that provide the data of the extracted portions 250 in the same data format as the original dataset 1502 (e.g., NetCDF4).

[0150] Improved techniques for managing multi-dimensional data have been described. The techniques include providing an original dataset 1502 that includes data arranged along a plurality of dimensions, each dimension covering its own original range in units of the dimension. The techniques further include extracting a plurality of portions 250 of the data from the original dataset 1502, each portion 250 extending over a reduced range of units of the dimension that is smaller than the original range in at least one dimension, and all of the extracted portions 250 together covering the original ranges of the original dataset 1502 in all dimensions.

[0151] While specific embodiments have been described, numerous alternative embodiments or variations are possible. For example, embodiments have been described in relation to NetCDF4 and HDF5, but these are merely examples. As technology evolves, further versions are expected to emerge, but the content described herein is not limited to the current versions only. Previous versions can also be used. Furthermore, the techniques presented herein can also be applied to datasets constructed in other formats. The present disclosure is not limited to a specific format or version.

[0152] Furthermore, features have been shown and described with reference to specific embodiments herein, but such features can be included in any of the disclosed embodiments and their variations and are included herein. Thus, it is understood that features disclosed in relation to any one embodiment are included in any other embodiment.

[0153] Furthermore, an improvement or a part thereof can be embodied as a computer program product including one or more non-transitory computer-readable storage media such as magnetic disks, magnetic tapes, compact discs, DVDs, optical discs, flash drives, solid state drives, SD (Secure Digital) chips or devices, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and / or the like (shown as an example as medium 1250 in FIGS. 12 and 22). Any number of computer-readable media can be used. The media may be encoded with instructions to perform the processes or operations described herein when executed on one or more computers or other processors. Such media can be regarded as a manufactured article or a machine and is transportable from one machine to another.

[0154] As used throughout this specification, the terms "comprising," "including," "containing," and "having" are intended to define something in an open-ended manner with respect to any particular item, step, element, or aspect. Also, as used in this specification, unless otherwise stated to the contrary, the term "set" means one or more of something. This is the same regardless of whether a singular or plural object follows the phrase "set of" or whether a singular or plural verb is conjugated. Also, an element that is "a set of" can represent a number less than all of the elements that exist. Thus, there may further exist elements of the same kind that are not part of the set. Further, ordinal expressions such as "first," "second," "third," etc. may be used as adjectives in this specification for purposes of identification. Unless otherwise specified, these ordinal expressions do not imply order or sequence. For example, a "second" event can occur before or after a "first" event, or even if the first event does not occur. Also, in this specification, if a particular element, feature, or operation is identified as being "a first" such element, feature, or operation, it should not be construed as being required that a "second" or other such element, feature, or operation must also exist. Rather, there may be cases where the "first" item is the only one. Also, unless otherwise stated to the contrary, "based on" is intended to be non-exclusive. Thus, "based on" should be construed to mean "at least partially based on" rather than "exclusively based on" unless otherwise indicated. Although specific embodiments are disclosed in this specification, these are provided by way of example only and should not be construed in a limiting sense.

[0155] Accordingly, those skilled in the art will understand that various changes in form and detail may be made to the embodiments disclosed herein without departing from the scope of the following claims.

Claims

1. Providing an original data set including data arranged along a plurality of dimensions of an N-dimensional space, each dimension of the N-dimensional space having a respective original range in dimension units, and the original data set having a data format; Extracting a plurality of data portions from the original data set, each portion extending over a reduced range of dimension units in at least one dimension of the N-dimensional space, and the extracted portions together covering all the original ranges of the N-dimensional space; Rendering the extracted portions in respective segments that provide the data of the extracted portions in the same data format as the original data set; A method for managing multi-dimensional data, comprising.

2. The method according to claim 1, wherein extracting the data portions includes defining portions having a dimension size of 1 in at least one dimension of the plurality of dimensions.

3. Placing each of the segments on each node of a plurality of computing nodes of a cluster; Tracking, in the object metadata of the cluster, the position of each of the segments on the nodes; The method according to claim 1, comprising.

4. The method according to claim 3, comprising reconstructing the original data set from each of the segments.

5. The original data set has an original header stored in the storage cluster, Reconstructing the original data set is Extracting data from each of the segments; Feeding the data extracted from each of the segments into a template data set; Taking out the original header from the storage cluster; Copying the original header to the template data set; The method according to claim 4, comprising.

6. The method according to claim 3, comprising tracking, in the object metadata, an association between a segment and each range of the plurality of dimensions covered by the segment.

7. The method according to claim 6, wherein the tracked association between the segment and each range identifies either (i) the exact range of the plurality of dimensions covered by the segment, or (ii) an inaccurate range that is not narrower than the exact range of the plurality of dimensions covered by the segment.

8. Receiving a query request for reading a set of data of the data set, wherein the query request identifies a set of predicates defining a region or a set of regions of the N-dimensional space; Accessing the object metadata, wherein the access searches, based on the association, for a set of candidate nodes identified as candidates for storing the set of data, and the set of candidate nodes is a subset of the plurality of computing nodes; Sending the query request or a modified version thereof to each of the set of candidate nodes and returning each share of the set of data; The method according to claim 6, comprising:

9. Receiving each share of the requested set of data from the set of candidate nodes; Merging each share to render a query result providing the whole of the set of data; The method according to claim 8, comprising:

10. The method according to claim 9, wherein at least one node of the set of candidate nodes returns an empty share that does not contain any of the set of data.

11. The method according to claim 8, wherein receiving the query request includes receiving a set of query conditions expressed as a set of dimensionless variables or ranges of labels, and the method includes converting the set of query conditions into the set of predicates.

12. Receiving an aggregation query request, the aggregation query request specifying a range of variable values stored in the data set at each coordinate of the N-dimensional space; Sending the aggregation query request or a modified version thereof to each or a subset of the plurality of computing nodes; Receiving partial aggregation query results from each or a subset of the plurality of computing nodes in response to the aggregation query or a modified version thereof being locally executed on each node; Combining the partial aggregation query results to provide an overall aggregation query result, according to the method of claim 3.

13. Receiving a processing request for performing an analysis procedure on the data set; Sending the processing request or a modified version thereof to each or a subset of the plurality of computing nodes; In response to the analysis procedure or a modified version thereof being executed locally on each node, receiving partial results from each of the plurality of computing nodes or a subset thereof; The method according to claim 3, wherein the partial results are combined to provide an overall analysis result.

14. The method according to claim 3, wherein the data set stores data adjacently in an array-based layout grouped by dimension, and extracting the plurality of data portions comprises, for each portion, selecting a sub-tensor of the multi-dimensional data within the data set such that it is included in each portion.

15. The method according to claim 3, wherein the data set stores the data in chunks having a uniform dimensional ratio, and extracting the plurality of data portions comprises, for each portion, selecting the chunks of the data set that form sub-tensors, each being an integer number greater than zero.

16. The method according to claim 15, wherein the extracted portions have a desired size, the chunks have a non-uniform data size, and selecting the integer number of chunks comprises, for at least one subset of the portions, selecting the number of each chunk having a combined data size that substantially matches the desired size.

17. The method according to claim 16, wherein selecting the integer number of chunks comprises selecting only physically contiguous chunks within the data set for inclusion in one portion.

18. The method according to claim 17, wherein the physically contiguous chunks include at least two chunks physically separated by non-chunk data.

19. Comprising a control circuit including a set of processors coupled to a memory, the control circuit being: Providing an original data set including data arranged along a plurality of dimensions of an N-dimensional space, each dimension of the N-dimensional space having a respective original range in dimension units, the original data set having a data format; Extracting a plurality of data portions from the original data set, each portion extending over a reduced range in dimension units in at least one dimension of the N-dimensional space, the extracted portions together covering all the original ranges of the N-dimensional space; A computerized device configured and arranged to render the extracted portion to each segment that provides the data of the extracted portion in the same data format as the original data set. [

20. ] A computer program product including a set of non-transitory computer-readable media having instructions that, when executed by a control circuit of a computerized device, cause the computerized device to execute a method for managing multi-dimensional data, the method comprising: Providing an original data set including data arranged along a plurality of dimensions of an N-dimensional space, each dimension of the N-dimensional space having a respective original range in units of dimensions, the original data set having a data format; Extracting a plurality of data portions from the original data set, each portion extending over a reduced range in units of dimensions in at least one dimension of the N-dimensional space, the extracted portions together covering all of the original ranges of the N-dimensional space; Rendering the extracted portions to each segment that provides the data of the extracted portions in the same data format as the original data set; A computer program product including the above.

Images