Performing time dynamic range partition transform

Abstract

The present disclosure provides a method (300) for performing time dynamic range partitioning, including receiving user data (12), the user data including partition keys (14) and cluster keys (16). The user data includes a respective total number of rows (18) defining a total data size of the user data. The method also includes identifying storage constraints (144) of a storage system (140). The storage constraints include a target file size and a target number of rows per file. The method further includes determining a plurality of split points (212) for the user data based on the storage constraints. The method also includes generating partition bucket numbers (222) from the plurality of split points, the partition bucket numbers defining ranges between each of the plurality of split points. The method further includes range partitioning each row of the user data into a file (214) using the partition bucket numbers.

Description

Technical Field

[0001] This disclosure relates to dynamic range partitioning of execution time. Background Technology

[0002] As cloud storage becomes increasingly popular, clustering and partitioning data layouts are being used more and more to reduce query costs and improve query performance. Because many tables are so large that they need to be split across many different servers, clusters of data blocks are often categorized by clustering keys to group related data in one place. Each data block includes a range of block values. When partitioning data into partitions and / or clustering structures, parameters are typically needed to define how the partitions are applied to the data. Unfortunately, partitioning parameters are often unknown until the partitioning process is executed; this can prevent or hinder the ability to dynamically store queryable data based on specific types of partitions. Summary of the Invention

[0003] One aspect of this disclosure provides a method for performing time-based dynamic range partitioning. The method includes receiving user data from a data storage system at data processing hardware. The user data includes a partition key, a clustering key, and a corresponding total number of rows defining the total data size of the user data. Each row of the user data is associated with a corresponding value defined by the partition key and includes one or more columns. The method also includes identifying storage constraints of the data storage system at the data processing hardware. Storage constraints include a target file size and a target number of rows per file. The method further includes determining multiple split points of the user data via the data processing hardware. The multiple split points are based on the corresponding total number of rows of the user data, the total data size of the user data, the target file size from the storage constraints, and the target number of rows per file from the storage constraints. The method also includes generating partition digits from the multiple split points via the data processing hardware. Partition digits define a range between each of the multiple split points. The method further includes partitioning each row range of the user data into files using the partition digits based on the corresponding value defined by the partition key, via the data processing hardware. The files store the user data and are configured to construct tables categorized according to the clustering key.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method includes receiving a data loading request from a user at a data processing hardware, the data loading request requesting the data storage system to range-partition an unknown number of future user data, and the received user data including the unknown number of future user data. Here, the data loading request may request the data storage system to use a clustering key to store the future user data. Optionally, the data loading request may occur at a data query system communicating with the data storage system, the data query system being configured to query the user data stored in the data storage system. The user data may correspond to a large amount of streaming user data that satisfies a dynamic range partitioning threshold, the dynamic range partitioning threshold indicating a minimum total data size.

[0005] In some configurations, partitioning each row of user data into a file using partition bits based on the corresponding value defined by the partition key includes generating empty partitions for any missing values, and when executing a query on the user data, identifying that the query includes the corresponding missing values, and excluding empty partitions from the read operations of the query. Here, the method may include receiving a maximum number of partitions for range partitioning at the data processing hardware, and determining, via the data processing hardware, that the number of corresponding non-empty partitions is less than the maximum number of partitions. Determining that the number of corresponding partitions is less than the maximum number of partitions may include generating a count of the multiple distinct values ​​defined by the partition key in the user data, and comparing the count of the multiple distinct values ​​defined by the partition key in the user data with the maximum number of partitions.

[0006] In some examples, storage constraints include a maximum number of partitions, and the method involves determining, via data processing hardware, whether the number of generated partition bits is less than the maximum number of partitions. In this example, when the number of generated partition bits is less than the maximum number of partitions, partitioning each row range of the user data into a file using the partition bits based on the corresponding values ​​defined by the partition key occurs.

[0007] In some implementations, the method includes receiving, at the data processing hardware, a data loading request from a data storage system requesting the data storage system to range-partition an unknown number of future user data entries, the received user data including the unknown number of future user data entries. In this implementation, the method includes receiving, at the data processing hardware, a maximum number of partitions for range partitioning, and, during the execution time of the data loading request, determining, via the data processing hardware, whether the number of generated partition bits is greater than the maximum number of partitions. In this implementation, when the number of generated partition bits is greater than the maximum number of partitions, it is not possible to perform range partitioning of each row of user data into a file using the partition bits based on the corresponding values ​​defined by the partition key.

[0008] Another aspect of this disclosure provides a system for performing execution-time dynamic range partitioning. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include receiving user data from a data storage system, the user data including a partition key, a clustering key, and a corresponding total number of rows defining the total data size of the user data, each row of the user data being associated with a corresponding value defined by the partition key and including one or more columns. These operations also include identifying storage constraints of the data storage system, these storage constraints including a target file size and a target number of rows per file. These operations further include determining a plurality of split points of the user data based on the corresponding total number of rows of the user data, the total data size of the user data, the target file size from the storage constraints, and the target number of rows per file from the storage constraints. These operations also include generating partition digits from the plurality of split points, these partition digits defining a range between each of the plurality of split points. These operations further include partitioning each row range of the user data into files using the partition digits based on the corresponding value defined by the partition key. The files store the user data and are configured to construct tables categorized according to the clustering key.

[0009] This aspect may include one or more of the following optional features. In some configurations, the operation includes receiving a data load request from a user of the data storage system, which requests the data storage system to range-partition an unknown number of future user data, the received user data including the unknown number of future user data. Here, the data load request may request the data storage system to use a clustering key to store the future user data. The data load request may occur at a data query system communicating with the data storage system, configured to query the user data stored in the data storage system. The user data may correspond to a large amount of streaming user data that meets a dynamic range partitioning threshold, which indicates a minimum total data size.

[0010] In some examples, using partitioning bits to range-partition each row of user data into a file based on the corresponding value defined by the partition key includes generating empty partitions for any missing values, and when executing a query on the user data, identifying that the query includes the corresponding missing values, and excluding empty partitions from the read operations of the query. The operation may include receiving a maximum number of partitions for range partitioning, and may include determining that the corresponding number of non-empty partitions is less than the maximum number of partitions. Determining that the corresponding number of partitions is less than the maximum number of partitions may include generating a count of the multiple distinct values ​​defined by the partition key in the user data, and may include comparing the count of the multiple distinct values ​​defined by the partition key in the user data with the maximum number of partitions.

[0011] In some implementations, the storage constraint includes a maximum number of partitions, and the operation includes determining whether the number of partition bits generated is less than the maximum number of partitions. In this implementation, when the number of partition bits generated is less than the maximum number of partitions, partitioning each row range of user data into a file using the partition bits based on the corresponding values ​​defined by the partition key occurs.

[0012] In some configurations, the operation includes receiving a data load request from a user of the data storage system, which requests the data storage system to range-partition an unknown number of future user data entries, and the received user data includes the unknown number of future user data entries. In this configuration, the operation includes receiving the maximum number of partitions for range partitioning, and determining, during the execution time of the data load request, whether the number of generated partition digits is greater than the maximum number of partitions. Further, when the number of generated partition digits is greater than the maximum number of partitions, it is not possible to perform range partitioning of each row of user data into a file using the partition digits based on the corresponding values ​​defined by the partition key.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings or in the following description. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an example data management environment.

[0015] Figures 2A to 2D yes Figure 1 A schematic diagram of an example data manager for a data management environment.

[0016] Figure 3 This is a flowchart illustrating an example of how data processing methods can be implemented.

[0017] Figure 4 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0018] The same reference symbols indicate the same elements in all figures. Detailed Implementation

[0019] Data storage systems can store user or client data in one or more large, queryable tables. The general structure of a table consists of data from individual records organized into rows. The length of a row can vary depending on the table's schema and / or the number of columns or fields associated with a particular record (i.e., a row). The table schema refers to the specified format of the table, which can define column names (e.g., fields), data types for specific columns, and / or other information. In some examples, the storage system is configured to generate a table schema based on the attributes of the user data it receives. For example, the storage system receives user data in a row-oriented format using a specific schema before retrieval. In other examples, the user or client coordinates with the storage system to define the schema for their user data before any user data is transferred to the storage system. Typically, when a data storage system receives user data, it retrieves that data by loading it into one or more files, which form the underlying structure that populates the table. Without further organization, the storage system loads the user data into files so that it can receive the user data. Here, the table format does not include any additional partitioning, grouping, or categorization formatting, unless there is further organization beyond how the user data is presented to the storage system. This type of table can be called a non-partitioned and non-clustered table. As a result, when a user wants to perform a query on a small portion of their data in the data storage system (e.g., a query with filtering conditions), the query may inevitably have to scan the entirety of the user's data in the storage system (i.e., the data in the table corresponding to the user's data).

[0020] While scanning all user data will generate an accurate response to the query, it still takes a significant amount of time, which can be reduced by formatting the user data in the storage system in a specific way. Since some data query systems include cost structures that charge based on the amount of data scanned, the incentive to reduce costs by minimizing the amount of data scanned during a query is increasingly strong. Therefore, for the sake of time and resource efficiency, data storage systems can be configured to organize user data in a more efficient query format. This is especially possible given the fact that users both submit data to the storage system and query his or her data; this combination allows users to coordinate or consolidate these efforts. Some examples of techniques for formatting data in a storage system include partitioning the data, arranging the data by distance, or some combination of both.

[0021] Table partitioning is a technique for dividing a large table of data into subsets of the table without creating a separate table for each subset. When data in a large table is partitioned into partitions, these partitions correspond to groups of rows that can be accessed and / or maintained individually. The advantage of partitioned tables is that they function as a single logical table when queried by a query system. When a storage system partitions data, it assigns each row to a partition based on one or more columns or schema elements in the table's data, called the partition key. Typically, the table is partitioned on the partition key relevant to the query. In other words, by using the partition key, which is frequently or always referenced when querying the data table, the query system can use the partition key as a filter to restrict access or reads to only relevant partitions (i.e., the relevant portion of the entire table). This technique, sometimes called partition pruning, improves query performance by eliminating the need for queries to read or scan other parts of the table. Additionally or alternatively, from a management perspective, partitioning allows flexibility so that administrators can manage partitioned tables collectively (e.g., for the entire table) or individually (e.g., for one or more portions of the table).

[0022] A partition key typically specifies a variable or value used to partition user data. Generally, a partition key can refer to any field (e.g., a column) corresponding to a row of data; however, certain forms of partitioning are more common due to the nature of queries. For example, queries often request data corresponding to a specific time (e.g., a date) or belonging to a specific time range (e.g., certain dates). Based on this frequent query format, some more common partitioning of user data includes partitioning by retrieval time (e.g., the loading time when the storage system loads the user data into storage or the arrival time of the user data in the storage system), by a date or timestamp other than the retrieval time (e.g., the data access time), or by an integer range. For example, using the retrieval time partitioning method, the storage system loads user data into files (e.g., automatically) based on date-based time units (e.g., hourly, daily, weekly, monthly, yearly, etc.). Here, the storage system identifies the retrieval time corresponding to each row of user data and loads the user data into the file corresponding to that specific retrieval time. In other words, when user data spans three days—Monday, Tuesday, and Wednesday—the storage system identifies four rows of user data corresponding to Monday (e.g., based on the retrieval date) and loads these four rows into the Monday file. It also identifies two rows of user data corresponding to Tuesday (e.g., based on the retrieval date) and loads these two rows into the Tuesday file, which is a different file from the Monday file. Finally, the storage system identifies six rows of user data corresponding to Wednesday (e.g., based on the retrieval date) and loads five of these six rows into the first Wednesday file, while the sixth row is loaded into a new second Wednesday file, since in this example, the file has a capacity of five rows. Because the table is formed from data blocks corresponding to the files, the table appears to have three subsets, one for each day, Monday, Tuesday, and Wednesday. Here, if a query requests user data whose retrieval date is Tuesday, due to the three subsets, the query can perform a fast lookup (e.g., based on metadata associated with the user data in the storage system) to identify the subset corresponding to Tuesday and then scan or read two of the 12 rows of user data in the Tuesday subset table. In other words, in this example, this form of partitioning reduces read operations to one-sixth of the user data. Column-based partitioning (e.g., based on date or timestamp) works similarly; partitioning is based on another time-based variable occurring, rather than the time of retrieval. For example, one or more columns in a row of user data could include a partitionable time field.

[0023] In some implementations, a partitioning function defines how data is partitioned on a partition key. In some examples, instead of defining which rows of data in a table are included in a partition group, the partitioning function identifies the boundary values ​​or split points between partitions. In other words, the actual number of partitions in the table equals the number of split points plus one. Range partitioning also uses a partition key, but this partitioning process identifies whether data is within the range specification of the partition key (i.e., belongs to that range specification). Here, range partitioning does not partition data when it is outside the range specification (e.g., does not belong to that range specification). Instead, when data is within the specified range of the range specification, range partitioning loads the data into files that generate a subset of tables corresponding to the specified range. For example, a range specification specifies that user data is partitioned into monthly ranges. Here, when user data corresponds to three months, range partitioning identifies each month as a split point or boundary of the given range. For example, when the three months are January, February, and March, the January range is from January 1 to January 31, the February range is from February 1 to February 28, and the March range is from March 1 to March 31. In this example, a partitioning function can identify the split point corresponding to the boundary of a range and / or whether the split point is inclusive or exclusive. For example, for a January range, the split point could be December 31st or January 1st. When the split point is December 31st, this date is in December and therefore is not included as part of the range from January 1st to January 31st. Therefore, the split point of December 31st would be an exclusive boundary point because it is not included in the range. On the other hand, the split point of January 1st would be an inclusive split point because the range from January 1st to January 31st includes that split point. To further illustrate this example, a partition key can identify a column in a row of user data as the partition value. In other words, if the partition key specifies a fetch time, the storage system performs range partitioning by identifying whether the fetch time of a row of user data falls within the range of January, February, or March. Using range partitioning, queries on range-partitioned tables can specify predicate filters based on the partition key (e.g., the partition column, such as fetch time) to reduce the amount of data scanned during the query.

[0024] Another method for formatting tables is clustering. In a clustered table, the data is organized based on the content of a clustering key. Here, a clustering key refers to one or more columns (e.g., in the table's schema) used to categorize the data (e.g., group related data together). When clustering occurs on multiple columns, the clustering key identifies the order of the columns that determine the categorization order of the data. When storage operations write data to a clustered table, the storage system categorizes the data using values ​​in the clustering columns and uses these values ​​to organize the data into multiple chunks within the storage system to form the clustered table. Utilizing clustered data (i.e., clustered user data tables), queries that filter user data based on clustering keys can eliminate the scanning of unnecessary data. For example, when a storage system loads data into a file, the file can include and be associated with metadata that identifies the minimum and maximum values ​​of the columns containing the user data in the loaded file. Using this information, when a query corresponds to clustered data, the query can first look for the metadata corresponding to the clustering key in the file and eliminate irrelevant files. For example, if a row of user data corresponds to transaction sales data, where one column defines the state (location) of the transaction, and the transaction sales data table is clustered by the state of the transaction, then a query for transactions in California can examine the file and determine from its metadata that there are no user data rows in the file corresponding to transactions in California. Here, this fast lookup prevents the query from having to perform further read operations on the file.

[0025] Unfortunately, without sufficient information to establish parameters for a partitioning technique beforehand, a particular partitioning technique has limited capabilities. This is especially true in the case of range partitioning. For example, range partitioning has traditionally been difficult to execute dynamically during the execution time of data loading. In other words, range partitioning typically relies on known priors, such as how many split points or range partitions are within the range specification. However, for user data being streamed to or batch-processed into a data storage system, it is usually unknown beforehand how much user data the storage system will receive for execution. Without knowing the size of the data received at the storage system before actually receiving the data, the storage system typically cannot determine the accurate split points (i.e., those that fit the actual data) that identify the range boundaries of the range partitions. These problems become further complicated when the data storage system is subject to constraints related to file size or the number of rows of data a file can store. In other words, the data storage system itself may have constraints that affect aspects of range partitioning. For example, if files can only be a certain size, efficient range partitioning should attempt to account for this size constraint to ensure that its range and / or split points do not result in files saturated due to range partitioning.

[0026] Figure 1An example of a data management environment (also referred to as a "data management system") 100 is illustrated. User device 110 associated with user 10 generates user data 12 during execution on its computing resources 112 (e.g., data processing hardware 114 and / or memory hardware 116). For example, user 10 uses an application operating on the data processing hardware 114 of user device 110 to generate user data 12. Because various applications are capable of generating large amounts of user data 12, user 10 typically utilizes other systems (e.g., remote system 130, storage system 140, or query system 150) for user data storage and / or user data management.

[0027] In some examples, user equipment 110 is a local device (e.g., associated with the location of user 10) that uses its own computing resources 112 and has the ability to communicate with one or more remote systems 130 (e.g., via network 120). Additionally or alternatively, user equipment 110 utilizes its access to remote resources (e.g., remote computing resource 132) to operate applications for user 10. User data 12 generated using user equipment 110 may initially be stored locally (e.g., in a data storage device 116 such as memory hardware 116) and then passed to or sent to remote system 130 via network 120 at creation time. For example, user equipment 110 uses remote system 130 to pass user data 12 to storage system 140.

[0028] In some examples, user 10 utilizes computing resources 132 of remote system 130 (e.g., a cloud computing environment) to store user data 12. In these examples, remote system 130 can receive user data 12 because it is generated by various user applications (e.g., streaming data). Here, a data stream (e.g., a stream of user data 12) refers to a continuous or generally continuous feed of data arriving at remote system 130 for storage and / or further processing. In some configurations, in addition to continuously streaming user data 12 to remote system 140, user 10 and / or remote system 130 configure user data 12 to be sent in batches at frequent intervals so that remote system 130 has a continuous supply of user data 12 for processing. Much like user equipment 110, remote system 130 includes computing resources 132, such as remote data processing hardware 134 (e.g., servers and / or CPUs) and memory hardware 136 (e.g., disks, databases, or other forms of data storage devices).

[0029] In some configurations, remote computing resource 132 is a resource utilized by various systems associated with and / or communicating with remote system 130. For example... Figure 1As shown, these systems may include storage system 140 and / or query system 150. In some examples, the functionality of these systems 140 and 150 may be integrated together in different permutations and combinations (e.g., built into each other) or may be independent systems capable of communicating with each other. For example, storage system 140 and query system 150 may be combined into a single system (e.g., as shown in the diagram). Figure 1 (These systems are shown by dashed lines in the diagram). Remote system 130 and its computing resources 132 may be configured to host one or more functions of these systems 140, 150. In some embodiments, remote system 130 is a distributed system, with its computing resources 132 distributed across one or more locations accessible via network 120.

[0030] In some examples, storage system 140 is configured to operate data warehouse 142 (e.g., a data repository and / or multiple databases) as a means of data storage for user 10 (or multiple users). Generally, data warehouse 142 stores data from one or more sources and can be designed to analyze, report on, and / or integrate data from its sources. Data warehouse 142 enables users (e.g., organizational users) to have a central storage repository and storage data access point. By including user data 12 in a central repository such as data warehouse 142, data warehouse 142 can simplify data retrieval for functions such as data analysis and / or data reporting (e.g., through an analytics system). Furthermore, data warehouse 142 can be configured to store large amounts of data so that user 10 (e.g., organizational users) can store large amounts of historical data to understand data trends. Because data warehouse 142 can be the primary or sole data storage repository for user data 12, storage system 140 may frequently receive large amounts of data (e.g., gigabytes per second, terabytes per second, or more) from user devices 110 associated with user 10. Additionally or alternatively, as storage system 140, storage system 140 and / or storage warehouse 142 may be configured for data security (e.g., data redundancy), for multiple users (e.g., multiple employees of an organization) from a single data source, and / or for simultaneous multi-user access. In some implementations, data warehouse 142 is persistent and / or non-volatile so that, by default, data is not overwritten or erased by new incoming data.

[0031] Query system 150 is configured to request information or data from storage system 140 in the form of query 160. In some examples, query 160 is initiated by user 10 as a request for user data 12 in storage system 140 (e.g., an export data request). For example, user 10 operates through query system 150 (e.g., an interface associated with query system 150) to retrieve user data 12 stored in data warehouse 142 of storage system 140. Here, query 160 can be user-initiated (i.e., directly requested by user 10) or system-initiated (i.e., configured by query system 150 itself). In some examples, query system 150 configures routines or recurring queries 160 (e.g., at a specified frequency) to allow user 10 to perform analysis or monitoring of user data 12 stored in storage system 140.

[0032] The format of query 160 can vary, but it may include a reference to specific user data 12 stored in storage system 150 and / or a request for user data 12 within a specific time period. For example, query 160 requests user data 12 from the previous seven days. In some configurations, user 10 sends user data 12 to storage system 140 in a specific format so that query system 150 can generate query 160 based on information about the specific format (e.g., using attributes of the format). For example, data storage system 140 receives user data 12 in a tabular format, where user data 12 populates the rows and columns of the table. Using the tabular format, user data 12 within the table may have rows and columns corresponding to a schema or header associated with user data 12. For example, user data 12 may refer to a business transaction performed by user 10. In this example, user data 12 may include columns for seller, buyer, transaction price, transaction quantity, and other transaction data collected by user 10 regarding their transaction. Here, each row may have a header or schema, such as a transaction number or identifier and / or a time entry associated with the transaction. Because storage system 140 can receive user data 12 in a specific format (e.g., a transaction table format), storage system 140 is configured to store user data 12 so that query system 150 can access elements of the format associated with user data 12 (e.g., relationships, headers, or other schemas) (e.g., providing further context or definition for user data 12). In other words, query system 150 generates query 160, which requests transaction prices for the previous seven days.

[0033] In response to query 160, query system 140 generates query response 162 that satisfies or attempts to satisfy the request of query 160 (e.g., a request for specific user data 12). Generally, query response 162 includes the user data 12 requested by query system 150 in query 160. Query system 150 may return this query response 162 to the entity that initiated query 160 (e.g., user 10) or another entity or system communicating with query system 150. For example, query 160 itself or query system 150 may specify that query system 150 deliver one or more query responses 162 to a system associated with user 10, such as an analytics system. For example, user 10 uses the analytics system to perform analysis on user data 12. Typically, query system 150 is configured to generate routine query 160 based on user data 12 in storage system 140, enabling the analytics system to perform its analysis (e.g., at a specific frequency). For example, query system 150 executes daily query 160 to extract transaction data from the most recent seven days for analysis and / or representation by the analysis system.

[0034] In some examples, query 160 corresponds to a query job. A query job refers to an operation / action performed by query system 150 on behalf of user 10. Some examples of actions performed by a query job include: loading user data 12 into storage system 140, exporting user data 12 from storage system 140, querying user data 12 from storage system 140, or copying user data 12 from storage system 140. Generally, a query job is first scheduled and then executed. For example, a query job regarding loading user data 12 into storage system 140 can be configured before query system 150 is actually able to transfer user data 12 to storage system 140 or coordinate the transfer of user data 12 to storage system 140. In other words, a query job can be set up so that query system 150 coordinates the transfer of user data 12 on a repetitive basis. For example, a query job specifies that user data 12 is transferred to storage system 140 at 5 p.m. every other day. In some implementations, query system 150 includes further parameters for the query job. For example, by requesting a work query to load data, query system 150 is configured to pass user data 12 to storage system 140 in either batch or streaming form. However, for either of these forms, query system 150 can be configured with a dynamic range partition threshold 152, which indicates the minimum total data size that user data 12 must exceed before query system 150 sends user data 12 to storage system 140 for storage processing.

[0035] In some implementations, when query system 150 receives input for query 160, query system 150 is configured to determine plan 154 to execute query 160. In other words, query 160 typically refers to a large table at a basic level, rather than a specific reference to the actual structure of a table in storage system 140. For example, query 160 simply specifies querying the tables of user data 12 in storage system 140 to derive transaction data for California over the past week. To abstract away from the more complex tables and / or storage structure of user data 12 in storage system 140, the input format of query 160 is simplified for use as a user interface. Therefore, user 10, executing or writing query 160, does not need to know the actual storage structure, but only the schema or fields of the high-level table structure to generate query 160. Query system 150, in conjunction with storage system 140, is able to decompose query 160 from user 10 and rewrite query 160 in a format that identifies potential operators on user data 12 to execute query 160 on the underlying structure of user data 12. That is, when query system 150 receives query 160, the system digests query 160 and plans how to execute query 160 on the actual structure of storage system 140. Such planning may require identifying a subset of tables (e.g., partitions) and / or files corresponding to the tables in query 160.

[0036] In some configurations, although query system 150 determines plan 154 before execution, plan 154 is still evolving. For example, query system 150 generates plan 154 during the planning phase of query 160, rather than during the execution of query 160. During execution, plan 154 may need to be adjusted to accommodate actual information present at execution time that was not included or is unavailable during planning. For example, when query 160 corresponds to a query job requesting to load data into storage system 140, the actual amount of user data 12 in such query job when user 12 generates query 160 is an unknown amount of future user data 12, as well as other unknowns about future user data 12, such as the actual size of user data 12 and / or the number of rows in user data 12. Based on these unknowns, query system 150 is configured to generate an adjusted plan 156 when executing query 160, such as a request to load data.

[0037] Reference Figure 1 and Figures 2A to 2DThe data management environment 100 also includes a manager 200. Manager 200 is configured to manage dynamic range partitioning. Here, dynamic range partitioning refers to range partitioning that occurs during the execution or runtime of a load operation on storage system 140. It is dynamic; in a sense, range partitioning occurs when user data 12 is actually loaded into storage system 140, therefore manager 200 must coordinate the generation of partitions (e.g., quantiles) for user data 12 while adapting to the constraints 144 of storage system 140 when retrieving user data 12 and storing it in file 224. Manager 200 can manage dynamic range partitioning by performing and / or coordinating operations related to systems 140 and 150 (e.g., storage operations and / or query operations) for user 10. Depending on the design of manager 200, its functionality can be centralized (e.g., residing in one of systems 140 and 150) or distributed among systems 140 and 150. In some examples, such as Figure 1 Manager 200 is configured to receive user data 12 from user 10 and facilitate storage operations at storage system 140. For example, manager 200 facilitates a data loading request from user 10. In response to the loading request from user 10, manager 200 acquires user data 12 and can convert user data 12 into a query-friendly format using range partitioning. Here, acquisition refers to obtaining user data 12 and / or importing user data 12 into storage system 140 (e.g., importing into data warehouse 142) to allow the system to use the acquired data (e.g., by querying the system). Generally, when manager 200 imports data, the data can be acquired in real time as it is emitted from the source (e.g., user 10 or user device 110 of user 10) or imported in batches, wherein manager 200 imports discrete blocks of data at periodic time intervals. During retrieval, manager 200 may validate the format of user data 12 (e.g., to conform to an acceptable format of storage system 140) and route user data 12 to data warehouse 142 (e.g., a specific data storage location in data warehouse 142 for user data 10). User data 12 may include partition key 14, cluster key 16, and one or more attributes 18 associated with user data 12.

[0038] like Figures 2A to 2D As shown, the manager 200 typically includes a splitter 210 and a partitioner 220. Here, the splitter 210 is configured to obtain the appropriate split point corresponding to the value of partition key 14. For example, when the value of partition key 14 refers to the acquisition time T... i At that time, the splitter 210 is based on the acquisition time T of user data 12. iSplit points 212 are generated for each distinct day. To generate split points 212, splitter 210 receives query 160, which identifies the loading request for user data 12 and the characteristics 18 of user data 12. In some configurations, such as... Figure 2A As shown, feature 18 identifies the total size 18, 18a of the user data 12 in load request 160 and the number of rows 18, 18b included in the user data 12 in load request 160. To accurately generate the split point 212, the splitter 210 also considers the constraints 144 of the storage system 140. For example, Figure 2A Storage system 140 is illustrated, which constrains the maximum size 144, 144a and / or the total number of lines 144, 144b of each file 224 in storage system 140. Without considering these constraints 144, splitter 210 can generate split points 212 that cause manager 200 (e.g., partitioner 220) to generate a range of user data 12 that is too large to store the files 224 of storage system 140. In other words, if the split points 212 generated by splitter 210 are too sparse, a large amount of user data 12 can be defined between two split points 212. To generate an accurate estimate of the split points 212, splitter 210 compares characteristics 18 of user data 12 (e.g., the total size 18a of user data 12 in load request 160 and / or the number of lines 18b included in user data 12) with the constraints 144 corresponding to the files 244 of storage system 140. For example, splitter 120 divides the total number of rows 18b in user data 12 by the number of rows 144b in each file 224 identified by constraint 144 of storage system 140. This division of the two numbers generates an estimate of the number of rows of user data 12 that may exist in a given file 224. In some embodiments, splitter 210 divides the total size 18a of user data 12 by the target file size 144a subject to storage constraint 144. Here, by dividing the total size 18a of user data 12 by the target file size 144a subject to storage constraint 144, splitter 210 generates an additional or alternative estimate of the number of files 224 that may be needed to store user data 12. Using one or both of these calculations, splitter 210 determines one or more split points 212 of user data 12 and passes these split points 212 to partitioner 220.

[0039] In some configurations, based on split point 212, partitioner 220 is configured to generate partition 222 or partition number of user data 12. Here, partition number defines the range between each adjacent split point 212 among a plurality of split points 212. Figure 2A As shown, partitions 222, 222a to 222d are based on the column corresponding to partition key 14 (for example, displayed as the date T is retrieved). iFor each row of user data 12 in the partition bits between split points 212, the partitioner performs range partitioning for each row by loading the row into file 224. In other words, partitioner 220 loads the range of values ​​together into file 224 based on partition key 14. Generally, partitioner 220 fills file 224 with user data 12 until file 224 reaches its capacity. Once file 224 reaches its capacity, partitioner 220 initiates a new file 224 and continues to fill the new file 224 with the user data 12 corresponding to the single partition 222 in the same way as the previous file 224.

[0040] In some implementations, it is common for user data 12 to correspond to more than one day (see, for example, see...). Figure 2C In these implementations, storage system 140 may be configured with an additional constraint that each file 224 includes only data from a single date (e.g., a single retrieval date). When this constraint exists in storage system 140, partitioner 220, when populating user data 12 for a specific partition 222, identifies when the timestamp associated with the data transitions from a first date to a second date (e.g., from day one to day two). When this occurs, partitioner 220 generates a new file 224 regardless of the capacity of the current file 224 being populated and loads the user data 12 with the timestamp of the second date into the new file 224, thus preventing a single file 224 from storing user data 12 with two different date timestamps. Based on this date separation technique of storage system 140, file 224 is inherently suitable for range partitioning.

[0041] In some configurations, when user data 12 includes one or more clustering keys 16 that identify one or more columns used to classify user data 12, partitioner 220 is also configured to range-partition user data 12. Generally, this combines partitioner 220 to partition user data 12 using multiple variables (e.g., partitioning variables for partitioning key 14 and one or more clustering variables 16 for clustering key 16). When this occurs, partitioner 220 may include operators that combine the variables of keys 14, 16 into a new structure. In some examples, partitioner 220 encodes this structure by combining multiple format variables into a string that has a string type as the underlying value of the structure. Partitioner 220 may also generate value operations for this new structure, such as Less(), IsComparable(), Equal(), AppentToString(), ParseFromString(), Copy(), Move(), and / or Memory(). By employing this new structure, partitioner 220 is able to generate partition values ​​(e.g., partition 222) for multivariate clustering. For example, partitioner 220 will... Figures 2A to 2D The example generates this structure because cluster key 16 identifies multiple cluster variables, columns C1 and C2.

[0042] Reference Figure 2B In some examples, query system 150 runs as a background process, actively receiving user data 12. As a background process, query system 150 can collect user data 12 until the amount of user data 12 meets a dynamic partitioning threshold 152. In other words, dynamic partitioning threshold 152 is configured to align processing resources to generate split points 212 and / or partitions 222. Query system 150 can apply dynamic partitioning threshold 152 to the batch loading or streaming loading of user data 12. Utilizing dynamic partitioning threshold 152, once the amount of user data 12 meets threshold 152 (e.g., exceeding a predetermined data amount), query system 150 can execute a request to load user data 12 into storage system 140 (e.g., using manager 200).

[0043] In some configurations, such as Figure 2C and Figure 2D Partitioner 220 is configured to perform quantile expansion or boundary injection. Quantile expansion is a technique that attempts to ensure that there is at least one partition 222 for each day (e.g., based on the retrieval date) so that no two rows of user data 12 with different date values ​​are mapped to the same partition 222. Figure 2C and Figure 2DAn example is illustrated where user data 12 corresponds to user data 12 for 5 days (e.g., shown as 4-16-2020, 4-17-2020, 4-18-2020, 4-19-2020, and 4-20-2020). Here, for the date 4-19-2020, no user data 12 exists. In other words, user data 12 jumps from the user data 12 corresponding to 4-18-2020 to 4-20-2020. In a normal splitting and partitioning process without quantile extension, splitter 210 typically does not generate a split point 212 corresponding to any boundary of date 4-19-2020 because no user data 12 exists for this date. In the absence of a boundary of date 4-19-2020, the partitioning process may have an increased possibility or risk of partitioning two rows with different date values ​​into the same partition 222. To avoid this risk, partitioner 220 performs quantile expansion by analyzing the date values ​​and determining which boundaries (e.g., split points 212, 212c to 212d) should be injected between row 6 R6 and row 7 R7. Here, this boundary injection generates an empty partition 226 as a placeholder for the date 4-19-2020. In some configurations, during the execution of a subsequent query 160 to read or export user data 12 within a date range that includes the empty partition 226, query 160 is configured to recognize the empty partition 226 and skip (i.e., exclude) it from any read operation of query 160.

[0044] Continue to refer to Figure 2D In some examples, storage system 140 additionally includes a constraint 144 on the maximum number of partitions 144, 144c that may occur in user data 12 at storage system 140 during dynamic partitioning. When storage system 140 includes such a constraint 144 regarding the maximum number of partitions 144c, storage system 140 can configure compliance with this constraint 144 in several different ways. For example, in some configurations, storage system 140 does not include any empty partitions 226 in the total number of partitions 222 in the partitioner. In other words, in this configuration, Figure 2C and Figure 2DThere are only four partitions 222a to 222d, even though there is an empty partition 226 in the stored user data 12. In another method, the manager 200 and / or storage system 140 determine whether the manager 200 (e.g., partitioner 220) violates the maximum number of partitions 144 by counting the total number of values ​​for partition key 14 present in the user data 12 and comparing that count with the number of partitions 222 generated by partitioner 220. The partitioning process violates constraint 144 on the maximum number of partitions 144dc when the total number of values ​​for partition key 14 is less than the count of partitions 222 (i.e., there are more partitions 222 than values). For example, when partition key 14 corresponds to a number of days, the manager 200 counts the total number of different days and evaluates whether this count of days is less than the maximum number of partitions 144. Figure 2D As shown, the manager 200 can be configured to determine whether it meets the maximum number of partitions 144c by comparing the count 228 of partitions 222 (e.g., total partitions 222, 226, or total non-empty partitions 222) or the count 228 of the value of partition key 14 identified by partitioner 220 with the maximum number of partitions 222. Figure 2D In this diagram, splitter 210 is shown performing this determination. When the partitioning process generates too many partitions 222, violating the maximum number of partitions 144c, query 160 for the load job fails, and dynamic partitioning does not occur for a given load request due to the violation of this constraint 144c. Conversely, when the number of partitions 222 meets the maximum number of partitions 144c, manager 200 is able to perform dynamic partitioning on user data 12.

[0045] Figure 3This is a flowchart illustrating an example layout of method 300, which dynamically partitions data during execution time. In operation 302, method 300 receives user data 12 from user 10 of data storage system 140. User data 12 includes a partition key 14, a clustering key 16, and constraints 18. Here, the constraints 18 of user data 12 include a corresponding total number of rows 18b defining the total data size 18a of user data 12. Each row of user data 12 is associated with a corresponding value defined by the partition key and includes one or more columns. In operation 304, method 300 identifies storage constraints 144 of data storage system 140. Storage constraints 144 include a target file size 144a and a target number of rows 144b per file 224. In operation 306, method 300 determines multiple split points 2212 for user data 12 based on the corresponding total number of rows 18b of user data 12, the total data size 18a of user data 12, the target file size 144a from storage constraint 144, and the target number of rows 144b of each file 224 from storage constraint 144. In operation 308, method 300 generates partitioning bits 222 from the multiple split points 212. Here, partitioning bits 222 define the range between each of the multiple split points 212. In operation 310, method 300 uses partitioning bits 222 to range-partition each row of user data 12 into files 224 based on the corresponding value defined by the partition key. Files 224 are configured to construct tables categorized according to clustering key 16.

[0046] Figure 4 This is a schematic diagram of an example computing device 400 that can be used to implement the systems and methods described herein. The computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions shown herein are merely exemplary and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0047] Computing device 400 includes a processor 410 (e.g., data processing hardware), a memory 420 (e.g., memory hardware), a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be mounted on a common motherboard or otherwise mounted where appropriate. Processor 410 can process instructions executed in computing device 400, including instructions stored in memory 420 or on storage device 430 for displaying graphical user interface (GUI) information on an external input / output device (such as a display 480 coupled to high-speed interface 440). In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and various types of memory, where appropriate. Similarly, multiple computing devices 400 can be connected to each device that provides the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0048] Memory 420 stores information non-temporarily in computing device 400. Memory 420 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporary memory 420 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0049] Storage device 430 provides massive storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices in storage area networks or other configurations. In additional embodiments, the computer program product is tangibly embodied as an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0050] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory hardware 420, display 480 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 450 that can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.

[0051] The computing device 400 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 400a or multiple times in a set of such servers 400a, or as a laptop computer 400b or as part of a rack server system 400c.

[0052] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system, including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit instructions and data to the storage system, at least one input device, and at least one output device.

[0053] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level programming and / or object-oriented programming languages ​​and / or assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0054] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). By way of example, processors suitable for executing computer programs include not only general-purpose and special-purpose microprocessors, but also any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or both of these mass storage devices. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROM disks and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0055] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback, and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0056] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method comprising: User data is received from the user of the data storage system at the data processing hardware. The user data includes a partition key, a cluster key, and a corresponding total number of rows that define the total data size of the user data. Each row of the user data is associated with a corresponding value defined by the partition key and includes one or more columns. The storage constraints of the data storage system are identified by the data processing hardware, and the storage constraints include the target file size and the target number of lines in each file. The data processing hardware determines multiple split points of the user data based on the following: The corresponding total number of rows in the user data; The total data size of the user data; The target file size derived from the storage constraints; as well as The target number of lines from each file in the storage constraint; The data processing hardware generates partitioning digits from the plurality of partitioning points, wherein the partitioning digits define the range between each of the plurality of partitioning points. as well as The data processing hardware uses the partitioning bits to range-partition each row of the user data into a file based on the corresponding value defined by the partitioning key, the file storing the user data and configured to construct a table categorized according to the clustering key.

2. The method according to claim 1, further comprising, at the data processing hardware, receiving a data loading request from the user of the data storage system, the data loading request requesting the data storage system to perform range partitioning on an unknown number of future user data, wherein, The received user data includes the unknown number of future user data.

3. The method according to claim 2, wherein, The data loading request requests the data storage system to store the future user data.

4. The method according to claim 2, wherein, The data loading request occurs at a data query system that communicates with the data storage system, and the data query system is configured to query the user's data stored in the data storage system.

5. The method according to claim 1, wherein, The user data corresponds to a large amount of streaming user data that meets the dynamic range partitioning threshold, which indicates the minimum total data size.

6. The method according to claim 1, wherein, Using the partition index based on the corresponding value defined by the partition key to range-partition each row of the user data into a file includes: Generate an empty partition for any missing date; and During the query performed on the user data: The query is identified to include the corresponding missing values; and The empty partition is excluded from the read operation of the query.

7. The method of claim 6, further comprising: The maximum number of partitions for range partitioning is received at the data processing hardware. as well as The data processing hardware determines that the number of corresponding non-empty partitions is less than the maximum number of partitions.

8. The method according to claim 7, wherein, Determining that the number of corresponding non-empty partitions is less than the maximum number of partitions includes: Generate a count of the multiple distinct values ​​defined by the partition key in the user data; and The counts of the multiple distinct values ​​defined by the partition key in the user data are compared with the maximum number of partitions.

9. The method according to claim 1, wherein, The storage constraint further includes a maximum number of partitions, and the method further includes: The data processing hardware determines whether the number of partition bits generated is less than the maximum number of partitions, and Wherein, when the number of generated partition bits is less than the maximum number of partitions, the user data is partitioned into the file for each row range using the partition bits based on the corresponding value defined by the partition key.

10. The method according to any one of claims 1 to 9, further comprising: At the data processing hardware, a data loading request is received from the user of the data storage system. The data loading request requests the data storage system to perform range partitioning on an unknown number of future user data, wherein the received user data includes the unknown number of future user data. The maximum number of partitions for the range partitioning is received at the data processing hardware; and During the execution time of the data loading request, the data processing hardware determines whether the number of generated partition bits is greater than the maximum number of partitions. Specifically, when the number of generated partition bits is greater than the maximum number of partitions, it is impossible to partition each row range of the user data into the file using the partition bits based on the corresponding value defined by the partition key.

11. A system comprising: Data processing hardware; as well as A memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: User data is received from the user in the data storage system. The user data includes a partition key, a cluster key, and a corresponding total number of rows that define the total data size of the user data. Each row of the user data is associated with a corresponding value defined by the partition key and includes one or more columns. Identify the storage constraints of the data storage system, including the target file size and the target number of lines per file; Multiple split points for the user data were determined based on the following: The corresponding total number of rows in the user data; The total data size of the user data; The target file size derived from the storage constraints; and The target number of lines from each file in the storage constraint; Generating partitioning digits from the plurality of partitioning points, wherein each partitioning digit defines a range between each of the plurality of partitioning points; and The user data is range-partitioned into a file that stores the user data and is configured to construct a table categorized according to the clustering key, using the partitioning digits based on the corresponding values ​​defined by the partitioning key.

12. The system according to claim 11, wherein, The operation further includes receiving a data loading request from the user of the data storage system, the data loading request requesting the data storage system to perform range partitioning on an unknown number of future user data, wherein the received user data includes the unknown number of future user data.

13. The system according to claim 12, wherein, The data loading request requests the data storage system to use the clustering key to store the future user data.

14. The system according to claim 12, wherein, The data loading request occurs at a data query system that communicates with the data storage system, and the data query system is configured to query the user's data stored in the data storage system.

15. The system according to claim 11, wherein, The user data corresponds to a large amount of streaming user data that meets the dynamic range partitioning threshold, which indicates the minimum total data size.

16. The system according to claim 11, wherein, Using the partition index based on the corresponding value defined by the partition key to range-partition each row of the user data into a file includes: Generate an empty partition for any missing date; and During the query performed on the user data: The query is identified to include the corresponding missing values; and The empty partition is excluded from the read operation of the query.

17. The system according to claim 16, wherein, The operation further includes: Receive the maximum number of partitions for range partitioning; and Determine that the number of corresponding non-empty partitions is less than the maximum number of partitions.

18. The system according to claim 17, wherein, Determining that the number of corresponding non-empty partitions is less than the maximum number of partitions includes: Generate a count of the multiple distinct values ​​defined by the partition key in the user data; and The counts of the multiple distinct values ​​defined by the partition key in the user data are compared with the maximum number of partitions.

19. The system according to claim 11, wherein, The storage constraint further includes a maximum number of partitions, and the operation further includes: Determine whether the number of partition digits generated is less than the maximum number of partitions, and Wherein, when the number of generated partition bits is less than the maximum number of partitions, the user data is partitioned into the file for each row range using the partition bits based on the corresponding value defined by the partition key.

20. The system according to any one of claims 11 to 19, wherein, The operation further includes: The user receives a data loading request from the data storage system, the data loading request requesting the data storage system to partition an unknown number of future user data into ranges, wherein the received user data includes the unknown number of future user data; Receive the maximum number of partitions for the range partitioning; and During the execution time of the data loading request, it is determined whether the number of generated partition bits is greater than the maximum number of partitions. Specifically, when the number of generated partition bits is greater than the maximum number of partitions, it is impossible to partition each row range of the user data into the file using the partition bits based on the corresponding value defined by the partition key.

Images