Adaptive sharding export method and system based on data density feedback

Abstract

The application relates to a self-adaptive sharding export method and system based on data density feedback, which acquires a query time range of data to be exported, determines a query time window by using a starting time of a current time slice and a sharding granularity, submits a sharding query, acquires an actual matching total and a data list, determines a density ratio according to the actual matching total and a target query upper limit, adjusts the sharding granularity of a next time slice according to the density ratio, the sharding granularity does not increase when the density ratio is larger, and the sharding granularity is constrained between a preset lower limit and an upper limit, the starting time of the next time slice is determined by using a record time field of a last record in the data list, and when the data list is empty, the ending time of the query time window is used for progression, and after a loop until a termination condition is met, data of each time slice is aggregated and output. The application makes the sharding granularity adaptively converge with data density, reduces memory occupation in a dense period and empty query rounds in a sparse period.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to an adaptive fragmentation export method and system based on data density feedback. Background Technology

[0002] With the continuous growth of data volume in information systems, batch exporting records from databases or search engines has become a common business requirement. When the amount of data exported in a single session reaches millions or even tens of millions of records, a single query will consume a lot of memory and exceed the single response capacity of the data interface. Therefore, the entire query is usually divided into multiple shards based on criteria such as time, and each shard is queried one by one before being aggregated into a complete export result. The way each shard is divided determines the amount of data covered by a single shard. If the amount of data in a single shard is too large, it will consume too much memory or trigger an interface timeout; if it is too small, it will generate too many query rounds.

[0003] In real-world business scenarios, data is often not evenly distributed over time. For example, in logistics and transactions, the data volume during promotional events can be several times to more than ten times that of normal days, and there are also significant differences in data volume between weekdays and non-weekdays. This fluctuating data density over time makes it difficult to predict in advance the number of records covered by each shard in the sharding export process, thus demanding a high degree of adaptability from the sharding method to changes in data density.

[0004] Existing sharding export methods, whether they involve dividing the query interval into equal-length segments with a fixed time granularity, pre-allocating data based on total statistics and then a fixed number of rows, advancing sequentially with a cursor based on a fixed number of rows, or manually pre-configuring the sharding granularity for each time period based on experience, all determine the basis for each shard statically before the query, without adjusting it based on the actual data density returned by the previous shard query. Therefore, when data is not uniformly distributed over time, the return volume of a single shard surges during data-dense periods, easily consuming excessive memory or triggering interface timeouts. Conversely, during data-sparse periods, a large number of nearly empty query rounds are generated, dragging down the overall export time due to the shard with the largest return volume. Furthermore, once the return volume of a shard exceeds expectations, subsequent shards continue to be queried at the original shard granularity, causing performance degradation to accumulate sequentially along the sharding order, making it difficult to adaptively converge to a shard granularity that matches the data density during the export process. Summary of the Invention

[0005] To address the problem that existing sharding export methods statically determine the sharding criteria before each query and do not adjust based on the actual data density returned by the previous shard, leading to an imbalance in the amount of data returned by each shard when the data is not uniformly distributed over time, excessive memory consumption during dense periods and a large number of near-empty queries during sparse periods, this application provides an adaptive sharding export method and system based on data density feedback.

[0006] Firstly, this application provides an adaptive fragmentation export method based on data density feedback, which adopts the following technical solution:

[0007] An adaptive sharding export method based on data density feedback is proposed, which obtains the query time range of the data to be exported and shards the data within the query time range for export according to the following steps:

[0008] S1. Determine the query time window based on the start time and granularity of the current time slice, submit a granular query request with the target query limit to the data query engine, and obtain the actual total number of matches and the data list for the current time slice;

[0009] S2. Determine the density ratio based on the actual total number of matches and the upper limit of the target query. Adjust the granularity of the next time slice based on the density ratio. The larger the density ratio, the smaller the granularity of the adjusted slice will not be increased. The adjusted granularity will be constrained between the preset lower limit and upper limit of granularity.

[0010] S3. Determine the start time of the next time slice based on the record time field of the last record in the data list; in response to an empty data list, keep the granularity of the slice unchanged, and determine the start time of the next time slice by adding one time unit to the end time of the query time window;

[0011] S4. In response to the failure to meet the preset termination condition, return to S1 to continue performing the sharded query for the next time slice until the preset termination condition is met, then aggregate the data list of each time slice and output it.

[0012] By adopting the above technical solution, the actual data density returned by each shard query is used as a feedback signal to determine the shard granularity of the next shard. This allows the shard granularity to be automatically narrowed during periods of high data density to suppress the amount of data returned per shard, and the shard granularity to be automatically widened during periods of low data density to reduce the number of query rounds. The shard granularity adaptively converges between the preset lower and upper limits to a value that matches the actual data density. At the same time, the start time of the next shard is determined by the actual time boundary of the returned data rather than a fixed window step size, so that adjacent shards are connected end to end in terms of the actual data distribution. This avoids omissions and near-empty queries caused by fixed step sizes in data gaps. Thus, even when the data is not uniformly distributed over time, the load of each shard can still be balanced and the overall memory usage and number of query rounds can be reduced.

[0013] Optionally, obtain the expected total amount of data to be exported. S1 to S4 are executed when the expected total amount of data is greater than the preset direct output threshold.

[0014] By adopting the above technical solution, the data will not enter the sharding loop when the data volume is small, thus avoiding the sharding scheduling overhead from exceeding the processing cost of the data itself. Adaptive sharding will only be enabled when the data volume exceeds the preset direct output threshold.

[0015] Optionally, S2 includes S2a and S2b in parallel. S2a. In response to a density ratio exceeding a preset multiple threshold, the granularity of the subdivision is divided by the density ratio and rounded down. The larger of the calculated result and the lower limit of the granularity is taken as the adjusted granularity. S2b. In response to a density ratio not exceeding a preset multiple threshold, the granularity is increased by a preset expansion factor. The smaller of the calculated result and the upper limit of the granularity is taken as the adjusted granularity.

[0016] By adopting the above technical solution, the density ratio is used as a scaling factor to adjust the granularity of the fragment inversely proportional to the density. This makes the granularity narrower more significantly as the single fragment return amount increases, thus accelerating the convergence of the granularity of the fragment to the matching value.

[0017] Optionally, after the shard query for each time slice is completed, the start time of the next time slice and the adjusted shard granularity are associated and stored as a breakpoint record, and the breakpoint record is set with a preset validity period; in response to receiving an export request carrying a query identifier and the existence of a breakpoint record associated with the query identifier, S1 is executed after using the start time and shard granularity in the breakpoint record as the initial values ​​of the start time and shard granularity of the current time slice in S1.

[0018] By adopting the above technical solution, the breakpoint record saves both the query progress and the adjusted sharding granularity. This ensures that when the export is resumed after an interruption, it not only continues the query position but also continues the converged sharding granularity, avoiding re-convergence from the initial granularity.

[0019] Optionally, the preset termination conditions include at least one of the following: the total amount of data collected reaches the preset export limit; the start time of the next time slice reaches the end time of the query time range; the cumulative number of executions of the sharded query reaches the preset limit.

[0020] By adopting the above technical solution, the loop exit is defined by three conditions: data full, interval traversal completed, and cumulative count limit. The cumulative count limit provides a safety net for abnormal situations, preventing the segmented loop from continuously spinning when it cannot terminate naturally.

[0021] Optionally, the method for determining the end time of the query time window in S1 is as follows: add the start time of the current time slice to the duration corresponding to the slice granularity and subtract one time unit to obtain the candidate end time of the window, and take the smaller of the candidate end time of the window and the end time of the query time range as the end time of the query time window.

[0022] By adopting the above technical solution, the window span is defined by the segmentation granularity corresponding to the duration and a time unit is subtracted to ensure that the beginning and end of adjacent windows do not overlap. At the same time, the last window is capped by the end time of the query time range to prevent the query from exceeding the time interval to be exported.

[0023] Optionally, in S1, the density gradient of the current time slice is estimated from the time distribution of the record time field of each record in the data list. The density gradient represents the trend of data density change over time within the current time slice.

[0024] By adopting the above technical solution, density gradients are extracted from the data distribution within a single piece, providing finer density information for piece granularity adjustment than the density ratio of a single piece.

[0025] Optionally, the boundary of the query time window for the next time slice can be corrected based on the density gradient feedforward, so that the query time window narrows accordingly when the density gradient indication becomes denser in the later part.

[0026] By adopting the above technical solution, the next segment window is narrowed predictively based on the density change trend, and the post-feedback adjustment is supplemented by pre-feedback correction.

[0027] Optionally, the deviation between the granularity obtained by feedforward correction and the granularity obtained by adjustment based on density ratio feedback is recorded. In response to the deviation continuously exceeding the preset deviation threshold, the weight of feedforward correction is reduced until it reverts to adjustment based solely on density ratio feedback.

[0028] By adopting the above technical solution, the accuracy of feedforward prediction is monitored by the deviation between the feedforward value and the feedback value. When the prediction is continuously inaccurate, the feedforward effect is automatically weakened to avoid the accumulation of feedforward error.

[0029] Optionally, a hysteresis rule is used to adjust the granularity of the fragments. When the density ratio exceeds a preset upper threshold, the granularity of the fragments is reduced. When the density ratio is lower than a preset lower threshold, the granularity of the fragments is expanded. When the density ratio is between a preset lower threshold and a preset upper threshold, the granularity of the fragments remains unchanged.

[0030] By adopting the above technical solution, the hysteresis band formed by the upper and lower thresholds is used to suppress the repeated scaling of the granularity of the slice when the density ratio fluctuates near the threshold, thereby reducing the ineffective oscillation of the granularity of the slice.

[0031] Optionally, in response to multiple consecutive time slices having empty data lists, a location query is submitted to the data query engine to determine the most recent existing record with data that is later than the start time of the current time slice, and the start time of the next time slice is determined by the record time field of the most recent record.

[0032] By adopting the above technical solution, when the continuous empty chip indicator enters the data gap area, a single location query can directly jump to the next time point where data exists, avoiding window-by-window empty query of the gap area.

[0033] Optionally, maintain the remaining quota to be collected. In response to the remaining quota to be collected being less than the target query limit, use the remaining quota to be collected as the target query limit when submitting a shard query request in this time slice.

[0034] By adopting the above technical solution, the remaining quota is used to limit the query limit of this film when the total export limit is approaching, thus avoiding over-collection of the last film.

[0035] Optionally, the breakpoint record is also associated with the density ratio before the storage interruption. In response to the query to resume based on the breakpoint record, the fragment granularity in the breakpoint record is corrected according to the stored density ratio before S1 is executed.

[0036] By adopting the above technical solution, during recovery, the density ratio before the interruption is combined to determine whether the existing fragment granularity is still applicable, and it is corrected when the density has changed, so as to avoid using the mismatched fragment granularity.

[0037] Optionally, a composite cursor can be constructed using the record time field of the last record and the primary key of the last record. The composite cursor can be used to determine the query start condition for the next time slice, so that multiple records with the same record time field are not missed and are not queried repeatedly.

[0038] By adopting the above technical solution, a composite cursor consisting of a time field and a primary key is used to define the sharding boundary, thus solving the problem of boundary omission and duplication when there are multiple records with the same time field.

[0039] Optionally, in response to the density ratio of multiple consecutive time slices falling into a preset stable range, the granularity of the slice is fixed and the remaining time range is traversed according to the fixed granularity. In response to the density ratio of subsequent time slices deviating from the preset stable range, the granularity of the slice is adjusted according to the density ratio.

[0040] By adopting the above technical solution, when the density ratio is continuously stable and the data distribution tends to be uniform, the granularity of the segments is fixed and the data is traversed quickly, reducing the computational overhead of adjusting the data segment by segment.

[0041] Optionally, the query results of the sharded query can be cached in the key-value cache and a second validity period can be set, which is shorter than the preset validity period of the breakpoint record.

[0042] By adopting the above technical solution, a short-term cache is set for the sharded query results, reducing repeated accesses to the data query engine by duplicate export requests.

[0043] Secondly, this application provides an adaptive fragmentation export system based on data density feedback, which adopts the following technical solution:

[0044] An adaptive fragmentation export system based on data density feedback includes:

[0045] The sharded query module is configured to obtain the query time range of the data to be exported. Within the query time range, the query time window is determined by the start time of the current time slice and the sharding granularity. The module submits a sharded query request with the target query upper limit to the data query engine and obtains the actual total matching volume and data list of the current time slice.

[0046] The density feedback adjustment module is configured to determine the density ratio based on the actual total number of matches and the upper limit of the target query, and adjust the granularity of the next time slice according to the density ratio. The larger the density ratio, the larger the adjusted granularity will be, and the adjusted granularity will be constrained between the preset lower limit and upper limit of granularity.

[0047] The boundary stepping module is configured to determine the start time of the next time slice based on the record time field of the last record in the data list, and in response to an empty data list, maintain the granularity of the slices and determine the start time of the next time slice by adding one time unit to the end time of the query time window; and

[0048] The loop control module is configured to trigger the sharded query module to continue executing sharded queries for the next time slice in response to the failure to meet the preset termination condition, until the preset termination condition is met, and then aggregate and output the data list of each time slice.

[0049] By adopting the above technical solution, the shard query module, density feedback adjustment module, boundary stepping module and loop control module work together to feed back the actual data density returned by each shard to the determination of the shard granularity, so as to realize the adaptive convergence of the shard granularity with the actual data density and the balance of the load of each shard.

[0050] In summary, this application includes at least one of the following beneficial technical effects:

[0051] 1. Use the actual data density returned by each shard as a feedback signal to determine the sharding granularity of the next shard, so that the sharding granularity adaptively converges between the preset lower and upper limits of granularity to a value that matches the actual data density. When the data is not uniformly distributed over time, the return amount of each shard is balanced, reducing the memory usage of a single shard during periods of high data density and the number of query rounds during periods of low data density.

[0052] 2. Determine the start time of the next shard based on the actual time boundary of the returned data, and advance the empty shards with the end time of the window. Combined with breakpoints to persist both query progress and shard granularity, as well as termination conditions of data fullness and count fallback, the shards are connected end to end in terms of actual data distribution, and can continue execution of the converged shard granularity after the export is interrupted.

[0053] 3. Further, the density gradient within a single piece is used to perform feedforward correction on the next piece window, and the feedforward weight is self-corrected based on the deviation between feedforward and feedback. Combined with the positioning jump in the empty piece area and the fixed granularity traversal after the density stabilizes, the accuracy of piece granularity adjustment is improved and the overhead of invalid queries and feedback is reduced under special working conditions such as drastic density changes and long gaps. Attached Figure Description

[0054] Figure 1 This is a flowchart of the adaptive fragmentation export method based on data density feedback in one embodiment of this application.

[0055] Figure 2 This is a schematic diagram of an adaptive fragmentation export system based on data density feedback in one embodiment of this application. Detailed Implementation

[0056] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0057] This application discloses an adaptive sharding export method based on data density feedback. This application uses the actual data density returned by each shard query as a feedback signal to adaptively adjust the sharding granularity of the next shard, exporting data in shards within the query time range. First, a query time window is determined based on the start time of the current time shard and the sharding granularity, and a shard query is submitted to obtain the actual total matching volume. Then, a density ratio is determined based on the actual total matching volume and the target query upper limit. The sharding granularity of the next time shard is adjusted based on the density ratio, and the start time of the next time shard is determined based on the actual data boundary. Finally, the process is repeated until a termination condition is met, at which point the data from each time shard is aggregated and output. This allows the sharding granularity to adaptively converge with the actual data density between a preset lower and upper limit, balancing the load on each shard when data is not uniformly distributed over time. The following provides a detailed description of the terminology, overall method flow, pre-processing splitting, breakpoint resumption, and system modules.

[0058] The following section will first explain some of the technical terms involved in the embodiments of this application.

[0059] The query time range refers to the overall search interval targeted by this export, defined by the start and end times. All data to be exported falls within this overall search interval.

[0060] A query time window refers to a retrieval interval defined for a single time slice within the query time range. It is determined by the start time and granularity of the time slice. A single slicing query is performed only within one query time window. The query time window differs from the query time range; the former is a local window for a single time slice, while the latter is a global interval covering all time slices. The two are not interchangeable.

[0061] Sharding granularity refers to the time span of the query time window for the next time slice. The larger the sharding granularity, the longer the time slice covers and the more records it may contain. The smaller the sharding granularity, the shorter the time slice covers and the fewer records it may contain.

[0062] Density ratio refers to the ratio of the actual total number of matches in a single time slice to the upper limit of the target query. The density ratio reflects the density of data within that time slice relative to the number of target query records. The higher the density ratio, the denser the data within that time slice.

[0063] The target query limit refers to the maximum number of records that can be returned in a single shard query, and is used to limit the amount of data returned in a single shard query.

[0064] Breakpoint records refer to the resume status saved during the export process. They include two dimensions: the start time of the next time slice and the current shard granularity. They are used to continue querying the position and the adjusted shard granularity when the export is resumed after an interruption.

[0065] The overall flow of this method is described in detail below with reference to the accompanying drawings. (Refer to...) Figure 1 This method, after obtaining the query time range of the data to be exported, exports the data in segments within the query time range, including steps S1 to S4, and executes them in a loop until the preset termination condition is met.

[0066] S1. Determine the query time window based on the start time and granularity of the current time slice. Submit a granular query request carrying the target query limit to the data query engine to obtain the actual total matching volume and data list for the current time slice. In the specific implementation, the end time of the query time window is determined by adding the start time of the current time slice to the duration corresponding to the granularity and subtracting one time unit to obtain the candidate end time of the window. Then, take the smaller of the candidate end time and the end time of the query time range as the end time of the query time window. Subtracting one time unit ensures that the query time windows of adjacent time slices do not overlap, and taking the smaller ensures that the last query time window does not exceed the query time range. For example, suppose the start time of the current time slice is [value], the granularity is [value] minutes, and the duration corresponding to the granularity is [value] seconds (3600 seconds), then the candidate end time of the window is [value] timestamps. If it is later than the end time of the query time range, then the end time of the query time range is used as the end time of the query time window. The data query engine can be, for example, the OpenSearch search engine or a relational database that supports time range filtering and pagination queries.

[0067] S2. Determine the density ratio based on the actual total number of matches and the target query limit. Adjust the sharding granularity of the next time slice based on the density ratio. A larger density ratio means the adjusted sharding granularity will not increase, and the adjusted sharding granularity will be constrained between a preset lower and upper limit. The density ratio is obtained by calculating the ratio of the actual total number of matches to the target query limit. The density ratio serves as a feedback signal representing the data density of the current time slice. Let represent the actual total number of matches and represent the target query limit. The density ratio is calculated as follows: where is the actual number of matched records returned by the query in the current time slice, and is the upper limit of the number of records returned by a single sharding query request. Set a multiplier threshold. Based on the comparison result between the density ratio and the multiplier threshold, adjust the sharding granularity through parallel steps S2a and S2b.

[0068] S2a. In response to a density ratio exceeding a multiple threshold, the sharding granularity is divided by the density ratio and rounded down. The larger of the calculated result and the granularity lower limit is taken as the adjusted sharding granularity to avoid excessive query rounds due to excessively small sharding granularity. Let represent the current sharding granularity, then the result of S2a is , where is the granularity lower limit.

[0069] S2b. In response to a density ratio not exceeding a multiple threshold, the granularity of the fragment is multiplied by a preset scaling factor. The smaller of the calculated result and the upper limit of granularity is taken as the adjusted granularity to avoid excessively large granularity leading to excessive single-fragment return. Let represent the scaling factor, then the result of S2b is , where is the upper limit of granularity.

[0070] For example, let's define the target query upper limit, initial shard granularity (minutes), multiplier threshold, expansion coefficient, lower granularity limit (minutes), and upper granularity limit (minutes). When the actual total number of matches in a certain time slice is less than the multiplier threshold of 2, the reduced shard granularity is set to minutes. When the actual total number of matches in a certain time slice is less than the multiplier threshold of 2, the expanded shard granularity is set to minutes. As another example, if the current shard granularity (minutes) and density ratio are less than the lower granularity limit of 5 minutes, the lower granularity limit of 5 minutes is used as the adjusted shard granularity to prevent the shard granularity from continuously shrinking to an unreasonable value. In practical implementation, a higher density ratio indicates denser data in the current time slice, and reducing the shard granularity of the next time slice according to the density ratio can suppress the single-shard return volume in the next time slice. A lower density ratio indicates sparser data, and expanding the shard granularity can reduce the number of query rounds in sparse segments. The lower and upper granularity limits constitute the adjustable range of the shard granularity, allowing the shard granularity to converge towards a value that matches the actual data density within this range.

[0071] The above describes one implementation of adjusting fragment granularity based on density ratio, namely, distinguishing between shrinking and expanding using a multiple threshold, and achieving scaling by dividing by the density ratio and multiplying by an expansion factor, respectively. In other embodiments, adjusting fragment granularity can also employ a hysteresis approach, i.e., setting an upper and lower threshold for granularity. The fragment granularity shrinks in response to a density ratio exceeding the upper threshold, expands in response to a density ratio below the lower threshold, and remains constant in response to a density ratio between the lower and upper thresholds, thereby suppressing repeated scaling of fragment granularity when the density ratio fluctuates around a single threshold. The difference from the previous approach is that the hysteresis approach uses a holding band formed by two thresholds instead of a single multiple threshold, reducing ineffective adjustments to fragment granularity. Both of these methods adjust fragment granularity according to the density ratio to adapt to the data density, and therefore both belong to implementations of adjusting fragment granularity based on density ratio.

[0072] In some embodiments, to obtain finer density information within a single time slice than a single density ratio, the density gradient of the current time slice is estimated from the temporal distribution of the record time field of each record in the data list. The density gradient characterizes the trend of data density change over time within the current time slice. Based on the density gradient, to supplement ex-post feedback with ex-ante prediction, the boundary of the query time window for the next time slice is corrected according to the density gradient feedforward, so that the query time window narrows accordingly when the density gradient indicates that the data in the later segment is denser, thereby responding to possible dense segments before querying the next time slice. Since the feedforward correction is based on historical density gradients and may be inaccurate when the density gradient changes abruptly, the deviation between the granularity obtained by the feedforward correction and the granularity adjusted according to the density ratio feedback is further recorded. In response to the deviation continuously exceeding a preset deviation threshold, the weight of the feedforward correction is reduced until it reverts to adjustment only according to the density ratio feedback, thus weakening the feedforward effect when the feedforward prediction is continuously inaccurate. In other embodiments, in response to the density ratio of multiple consecutive time slices falling into a preset stable range, the granularity of the sharding is fixed and the remaining query time range is traversed according to the fixed granularity. In response to the density ratio of subsequent time slices deviating from the preset stable range, the granularity of the sharding is adjusted according to the density ratio, thereby reducing the calculation of adjusting the granularity of the sharding piece by piece when the data distribution tends to be uniform.

[0073] S3. Determine the start time of the next time slice using the record time field of the last record in the data list. In response to an empty data list, maintain the granularity of the sharding and determine the start time of the next time slice by adding one time unit to the end time of the query time window. In practice, data is often not uniformly distributed along the time axis. Using the record time field of the last record in the data list plus one time unit as the start time of the next time slice ensures that the next time slice continues from the end of the actual data in the previous time slice, and adjacent time slices are connected end-to-end in terms of actual data distribution, reducing the number of time slices without records in sparse data areas. For example, if the record time field of the last record in the data list of a certain time slice is [value missing], then the start time of the next time slice is [value missing]. In response to an empty data list for a certain time slice, there is no last record available for retrieval within that time slice. In this case, maintain the granularity of the sharding and determine the start time of the next time slice by adding one time unit to the end time of the query time window for that time slice, thus skipping the empty time slice and continuing the search.

[0074] In some embodiments, to more quickly skip data gaps when there is no data in multiple consecutive time slices, in response to the data list being empty for multiple consecutive time slices, a location query is submitted to the data query engine to determine the most recent record with existing data that is later than the start time of the current time slice. The start time of the next time slice is then determined using the record time field of this most recent record, and the process jumps directly to the next time point with existing data. In other embodiments, to handle the case where multiple records correspond to the same record time field, a composite cursor is constructed using the record time field of the last record in the data list and the primary key of that last record. This composite cursor is used to determine the query start condition for the next time slice, ensuring that multiple records with the same record time field are neither missed nor queried repeatedly.

[0075] S4. In response to the failure to meet the preset termination condition, return to step S1 to continue executing the sharded query for the next time slice until the preset termination condition is met, then aggregate and output the data list of each time slice. In specific implementations, the preset termination condition includes at least one of the following: the total amount of data collected reaches the preset export total limit; the start time of the next time slice reaches the end time of the query time range; the cumulative execution count of the sharded query reaches the preset number of times limit. The first two conditions correspond to normal export completion, i.e., the required total amount of data has been collected or the entire query time range has been traversed; the cumulative execution count reaching the preset number of times limit provides a backup for abnormal situations, forcibly exiting the loop when the sharded query cannot terminate naturally due to an abnormality, avoiding continuous idle running. In some embodiments, to avoid collecting more data than needed in the last time slice when approaching the export total limit, the remaining quota to be collected is maintained. In response to the remaining quota to be collected being less than the target query limit, the remaining quota to be collected is used as the target query limit when submitting the sharded query request for that time slice, so that the collection volume of the last time slice is capped at the remaining quota to be collected.

[0076] The following describes the pre-processing before shard export. In some embodiments, before executing step S1, this method first obtains the expected total amount of data to be exported. Steps S1 to S4 are executed when the expected total amount of data is greater than a preset direct output threshold. If the expected total amount of data is not greater than the preset direct output threshold, the sharding loop is not entered; instead, a single direct output query is executed and the query result is output. If the expected total amount of data is greater than the preset direct output threshold, adaptive sharding in steps S1 to S4 is executed. For example, if the preset direct output threshold is 100,000 records, a single direct output query is executed directly when the expected total amount of data is 8,000 records; and adaptive sharding export in steps S1 to S4 is entered when the expected total amount of data is 5 million records. In specific implementations, a single query can complete the process when the data volume is small, while the scheduling overhead of the sharding loop is relatively large. Therefore, a preset direct output threshold is used to distinguish between single direct output and adaptive sharding. The above describes one implementation method that distinguishes between single direct output and adaptive sharding using a fixed preset direct output threshold. In other embodiments, the preset direct output threshold can also be dynamically determined based on the actual data volume statistics of recent similar export requests. That is, the direct output threshold is adjusted with reference to the data volume of several recent similar exports, so that the direct output threshold is adapted to the recent data volume level. Both of the above methods determine whether to enter adaptive sharding based on the comparison result between the expected total data volume and the direct output threshold, and therefore both belong to the implementation methods of pre-sharding.

[0077] The following describes the resuming of exports from interrupted points. In some embodiments, after the shard query for each time slice is completed, this method associates the start time of the next time slice with the adjusted shard granularity and stores it as a breakpoint record. The breakpoint record has a preset validity period. In response to receiving an export request carrying a query identifier and the existence of a breakpoint record associated with the query identifier, step S1 is executed after using the start time and shard granularity in the breakpoint record as the initial values ​​of the start time and shard granularity of the current time slice in step S1. The breakpoint record simultaneously saves two dimensions: the query position and the adjusted shard granularity. This ensures that when the export resumes after an interruption, it not only continues from the time position at the point of interruption but also continues from the shard granularity that was converged at the time of interruption, without needing to reconverge from the initial shard granularity. In specific implementations, breakpoint records are stored, for example, through cached key-value objects, such as Redis's RBucket. The key of a breakpoint record is a string in the form of `exportDetail:{queryId}`, and the value is a string in the form of `{startTime}_{stepMin}`. For example, the value 171861200_60 indicates that the start timestamp of the next time slice is 171861200, the shard granularity is 60 minutes, and the preset validity period of the breakpoint record is, for example, 1 minute. To further adapt to changes in data density during interruptions, in some embodiments, breakpoint records are also associated with the density ratio before the interruption. In response to a query that resumes based on the breakpoint record, the shard granularity in the breakpoint record is corrected according to the stored density ratio before step S1 is executed. This correction of the shard granularity when the data density has changed during the interruption avoids using a shard granularity that does not match the current data density after recovery. In other embodiments, the query results of sharded queries are cached in a key-value cache and a second validity period is set. The second validity period is, for example, 10 seconds, which is shorter than the preset validity period of the breakpoint record, thereby reducing repeated access to the data query engine when repeated export requests arrive.

[0078] The system for implementing the above method is described below. (Refer to...) Figure 2 The adaptive sharding export system based on data density feedback includes a sharding query module, a density feedback adjustment module, a boundary stepping module, and a loop control module.

[0079] The sharded query module is configured to obtain the query time range of the data to be exported. Within the query time range, it determines the query time window based on the start time of the current time slice and the shard granularity. It then submits a sharded query request carrying the target query upper limit to the data query engine and obtains the actual total matching volume and data list for the current time slice. The sharded query module performs the function corresponding to step S1 above.

[0080] The density feedback adjustment module is configured to determine the density ratio based on the actual total number of matches and the target query upper limit, and adjust the granularity of the next time slice according to the density ratio. The larger the density ratio, the smaller the adjusted granularity will be, and the adjusted granularity will be constrained between a preset lower limit and an upper limit. The density feedback adjustment module performs the function corresponding to step S2 above.

[0081] The boundary stepping module is configured to determine the start time of the next time slice based on the record time field of the last record in the data list. In response to an empty data list, it maintains the granularity of the slices and determines the start time of the next time slice by adding one time unit to the end time of the query time window. The boundary stepping module performs the function corresponding to step S3 above.

[0082] The loop control module is configured to trigger the sharded query module to continue executing sharded queries for the next time slice in response to the failure to meet the preset termination condition, until the preset termination condition is met, at which point the data lists of each time slice are aggregated and output. The loop control module performs the function corresponding to step S4 above.

[0083] Between the modules, the density feedback adjustment module receives the actual total matching amount obtained by the shard query module to determine the density ratio, the boundary stepping module receives the data list obtained by the shard query module to determine the start time of the next time slice, and the loop control module triggers the shard query module to enter the next time slice according to the preset termination condition. The modules work together to complete the shard export by sharing the actual total matching amount, data list and shard granularity.

[0084] In some embodiments, the system implementing the above method structurally includes a business layer, a data access layer, a caching layer, and a data engine layer. The business layer handles closed-loop processing including pre-judgment, breakpoint loading, sharded querying, density calculation, granularity adjustment, data boundary stepping, breakpoint persistence, and termination judgment. The data access layer encapsulates query calls to the data query engine and read / write operations to the cache, providing a unified data access interface for the business layer. The caching layer, for example, is a Redis cluster, storing breakpoint records and sharded query results. The data engine layer is the aforementioned data query engine, storing detailed data to be exported and responding to time range filtering and pagination query requests.

[0085] In some embodiments, this method runs in an environment including an application server, a cache server, and a data query engine server. The application server is configured, for example, with a 4-core processor, 8GB of memory, and 50GB of available disk space, to handle the business logic calculations for sharded export. The cache server is configured, for example, with a 2-core processor and 4GB of memory, to store breakpoint records. The configuration of the data query engine server is determined based on the data scale, for example, with an 8-core processor and 32GB of memory. In terms of the software environment, this method runs, for example, on Java 8 or later with the Spring Boot framework. The cache uses, for example, Redis 3.0 or later with the Redisson client for reading and writing, and a dedicated thread pool is configured to execute the sharded export task. This thread pool has, for example, a core thread count of 10, a maximum thread count of 500, and a queue capacity of 0, to accommodate high-concurrency query scenarios.

[0086] The same or similar parts in the various embodiments of this application can be referenced to each other, and the technical features in the various embodiments can be combined to form new embodiments without conflict.

[0087] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An adaptive fragmentation export method based on data density feedback, characterized in that, Obtain the query time range of the data to be exported, and export the data in chunks within the query time range according to the following steps: S1. Determine the query time window based on the start time and granularity of the current time slice, submit a granular query request carrying the target query limit to the data query engine, and obtain the actual total number of matches and the data list of the current time slice; S2. Determine the density ratio based on the actual total number of matches and the target query upper limit, adjust the granularity of the next time slice based on the density ratio, the larger the density ratio, the smaller the adjusted granularity will not be increased, and constrain the adjusted granularity between the preset lower limit and upper limit of granularity; S3. Determine the start time of the next time slice using the record time field of the last record in the data list; in response to the data list being empty, keep the granularity of the slice unchanged, and determine the start time of the next time slice by adding one time unit to the end time of the query time window; S4. In response to the failure to meet the preset termination condition, return to S1 to continue performing the sharded query for the next time slice until the preset termination condition is met, then aggregate the data list of each time slice and output it.

2. The adaptive fragmentation export method based on data density feedback according to claim 1, characterized in that, Obtain the expected total amount of data to be exported. Steps S1 to S4 are executed when the expected total amount of data is greater than a preset direct output threshold.

3. The adaptive fragmentation export method based on data density feedback according to claim 1, characterized in that, S2 includes S2a and S2b, which are parallel: S2a. In response to the density ratio exceeding a preset multiple threshold, the granularity of the segment is divided by the density ratio and rounded down, and the larger of the calculation result and the lower limit of the granularity is taken as the adjusted granularity of the segment. S2b. In response to the density ratio not exceeding the preset multiple threshold, the particle size of the segment is increased by a preset expansion factor, and the smaller of the calculation result and the upper limit of the particle size is taken as the adjusted particle size.

4. The adaptive fragmentation export method based on data density feedback according to claim 1, characterized in that, The method further includes: after the shard query of each time slice is completed, the start time of the next time slice is associated with the adjusted shard granularity and stored as a breakpoint record, the breakpoint record having a preset validity period; in response to receiving an export request carrying a query identifier and the existence of the breakpoint record associated with the query identifier, the method executes S1 after using the start time and shard granularity in the breakpoint record as the initial values ​​of the start time and shard granularity of the current time slice in S1.

5. The adaptive fragmentation export method based on data density feedback according to claim 1, characterized in that, The preset termination conditions include at least one of the following: the total amount of data collected reaches the preset upper limit of the total amount exported; the start time of the next time slice reaches the end time of the query time range; the cumulative number of times the sharded query is executed reaches the preset upper limit of the number of times.

6. The adaptive fragmentation export method based on data density feedback according to claim 1, characterized in that, The method for determining the end time of the query time window in S1 is as follows: add the start time of the current time slice to the duration corresponding to the slice granularity and subtract one time unit to obtain the candidate end time of the window, and take the smaller of the candidate end time of the window and the end time of the query time range as the end time of the query time window.

7. An adaptive fragmentation export system based on data density feedback, characterized in that, include: The sharded query module is configured to obtain the query time range of the data to be exported, determine the query time window within the query time range based on the start time of the current time slice and the sharding granularity, submit a sharded query request carrying the target query upper limit to the data query engine, and obtain the actual total matching volume and data list of the current time slice. The density feedback adjustment module is configured to determine the density ratio based on the actual total number of matches and the target query upper limit, adjust the granularity of the next time slice based on the density ratio, and the larger the density ratio, the larger the adjusted granularity will be, and constrain the adjusted granularity between the preset lower limit and upper limit of granularity. The boundary stepping module is configured to determine the start time of the next time slice based on the record time field of the last record in the data list, and in response to the data list being empty, to keep the granularity of the slice unchanged and to determine the start time of the next time slice by adding one time unit to the end time of the query time window. as well as The loop control module is configured to trigger the sharding query module to continue executing sharding queries for the next time slice in response to the failure to meet the preset termination condition, until the preset termination condition is met, and then aggregate and output the data list of each time slice.

Images