Massive data sharding aggregation query method and system based on hybrid database architecture

Abstract

This invention discloses a method and system for massive data sharding and aggregation query based on a hybrid database architecture, belonging to the field of data query technology. Upon receiving an aggregation query request, a consistency cross-section identifier is generated. Under the constraint of the consistency cross-section identifier, the target data is ordered sharded, and a two-sided compensation shard is constructed to cover the shard boundary data. Candidate shards are extracted in parallel and shard fingerprint information is generated to establish a mapping relationship between the basic shards and the two-sided compensation shards. An aggregation closure unit is constructed based on the grouping field summary. The target shard set is determined and deduplication rules are generated. Duplicate contributions are removed and boundary truncation compensation is performed on the local aggregation results to obtain a corrected result set. Finally, global aggregation is performed, and the query results corresponding to the consistency cross-section identifier are output. This invention can effectively solve the statistical bias problem caused by shard boundaries and data inconsistency during massive data sharding and aggregation, improving the accuracy and stability of the query results.

Description

Technical Field

[0001] This invention relates to the field of data query technology, specifically to a method and system for massive data sharding and aggregation query based on a hybrid database architecture. Background Technology

[0002] In scenarios such as financial ledger and real-time risk control, massive amounts of detailed data are usually stored in a hybrid database architecture (such as HTAP and embedded OLAP working together). When executing queries with multidimensional grouping, window functions and nested aggregations on cross-shard data, a rare but serious problem can easily occur: due to time slice differences and inconsistent columnar compression boundaries during the export and loading of shard data, the same logical group is split into multiple local aggregation units in different shards, resulting in implicit duplicate counts or omissions in the final aggregation stage. Moreover, this problem is difficult to detect through traditional consistency checks under high-concurrency short-cycle queries. The uncertainty of query routing further exacerbates the problem, leading to result drift and system resource fluctuations. Existing technologies lack a refined processing mechanism for this type of "shard semantic mismatch". Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for massive data sharding and aggregation query based on a hybrid database architecture, so as to solve the shortcomings of the background technology.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for massive data sharding and aggregation query based on a hybrid database architecture, comprising:

[0005] S100: Receive an aggregation query request, parse the target table, filter conditions, grouping fields, aggregation fields and time range, and generate a consistency profile identifier V for the submission version at the time the aggregation query request is received;

[0006] S200. Under the constraint of consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and the boundary records of the preset range are extracted for the start boundary and end boundary of each basic segment to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set.

[0007] S300: Extract each candidate segment in the candidate segment set in parallel and generate segment fingerprint information for each candidate segment.

[0008] S400. Establish the mapping relationship between the basic fragment and the bilateral compensation fragment based on the fragment number and fragment fingerprint information of each candidate fragment, and identify the associated records that cross the boundaries of adjacent basic fragments and have the same grouping key based on the grouping field summary, and construct the corresponding aggregation closure unit.

[0009] S500. Determine the target shard set to participate in this aggregation query based on the aggregation closure unit, and generate deduplication rules for each aggregation closure unit.

[0010] S600. Perform local aggregation calculation on the target fragment set to obtain fragment local results. Based on the deduplication and fragment fingerprint information, remove duplicate contribution values ​​in the fragment local results and compensate for boundary truncation contribution values ​​to obtain a complete closure correction result set.

[0011] S700. Perform global aggregation on the correction result set according to the grouping field and aggregation field, and output the final aggregation query result corresponding to the consistency section identifier V.

[0012] Preferably, the step of establishing the mapping relationship between the basic shard and the bilateral compensation shard based on the candidate shard number and shard fingerprint information includes: using the candidate shard number as a node, constructing a shard topology graph based on boundary semantic similarity; when two candidate shards satisfy the conditions that their primary key intervals are adjacent, their primary key intervals overlap, or the difference between their primary key intervals does not exceed 128, listing them as a pair of nodes to be connected; when the boundary semantic similarity of the pair of nodes to be connected is greater than or equal to 0.55, establishing a connection relationship in the shard topology graph, and determining the boundary direction based on the minimum and maximum primary key values; using the candidate shard number corresponding to the basic shard as the center node, selecting connection nodes with a boundary semantic similarity greater than 0.75 as bilateral compensation shard nodes, and determining the front bilateral compensation shard and the back bilateral compensation shard according to the boundary semantic similarity from high to low, and establishing a mapping relationship table.

[0013] Preferably, constructing the corresponding aggregation closure unit specifically includes: extracting the grouping field summary and boundary record summary of adjacent basic segments and their corresponding bilateral compensation segments, and restoring the grouping key distribution sequence in the boundary records according to the primary key order; concatenating the grouping key distribution sequence corresponding to the end boundary of the adjacent basic segment with the grouping key distribution sequence corresponding to the start boundary of the subsequent bilateral compensation segment, and determining the continuous interval as a candidate record segment when at least 3 consecutive records with the same grouping field value appear; performing primary key continuity condition and time order consistency condition judgment on the candidate record segment, and determining the candidate record segment that satisfies the primary key difference not greater than 1 and the time field value maintains a non-decreasing order as an associated record set; merging the associated record set according to the same grouping field value, continuous primary key interval and continuous time interval, and determining it as an aggregation closure unit after passing the record quantity consistency check, boundary record consistency check and grouping field value uniqueness check.

[0014] Preferably, step S100 specifically includes: parsing the aggregation query request, extracting the target table, filtering conditions, grouping fields, aggregation fields, and time range; reading the maximum visible commit version corresponding to the moment the aggregation query request is received; associating and encapsulating the maximum visible commit version with the target table identifier, filtering conditions, grouping fields, aggregation fields, and time range to generate a consistency profile identifier V.

[0015] Preferably, the fragment fingerprint information generated in S300 includes: fragment number, fragment type identifier, minimum primary key value, maximum primary key value, total number of records, boundary record summary, and grouping field summary; wherein, the boundary record summary is generated by encoding the primary key field value, grouping field value, and time field value of records near the start and end boundaries of the candidate fragment, and the grouping field summary is used to describe the distribution of each grouping key within the candidate fragment; for bilaterally compensated fragments, the fragment fingerprint information also includes the source base fragment number and boundary direction.

[0016] Preferably, the generation of deduplication attribution rules in S500 specifically includes: for each record within each aggregation closure unit, calculating the attribution priority score of the record in each shard based on the coverage range of the primary key field value in the base shard and the bilateral compensation shard; wherein, the attribution priority score is determined by a combination of the primary key position offset and the boundary distance, with the primary key position offset accounting for 60 parts and the boundary distance accounting for 40 parts; selecting the shard with the highest attribution priority score as the unique attribution shard of the record, and marking the remaining shards as redundant shards; generating a set of filtering conditions according to the shard number, the range of primary key field values, and the range of grouping field values ​​to form the deduplication attribution rules.

[0017] Preferably, step S600 specifically includes: grouping data records for each target fragment in the target fragment set according to the grouping field, and performing local aggregation calculation according to the type of the aggregation field to obtain fragment local results; performing record-level filtering on the original record set corresponding to the fragment local results according to the deduplication attribution rules, retaining only the record contribution values ​​in the uniquely attributed fragments, and removing duplicate contribution values ​​in redundant fragments to obtain preliminary deduplication results; identifying missing contribution intervals based on the minimum primary key value, maximum primary key value, and boundary record summary in the fragment fingerprint information, and extracting compensation records that meet the primary key range and time continuity conditions from the corresponding bilateral compensation fragments; merging the preliminary deduplication results with the compensation records, and performing aggregation calculation again according to the grouping field to obtain a complete closure correction result set.

[0018] Preferably, step S700 specifically includes: sorting all records in the correction result set according to the grouping field value, and dividing records with the same grouping field value into the same global group set; performing further aggregation calculations within each global group set based on the type of the aggregation field, and filtering records according to the time range and version upper bound corresponding to the consistency section identifier V; concatenating the grouping field value, aggregation field value, and consistency section identifier V corresponding to each global group set to generate a unique result identifier, and constructing a global result mapping relationship; and outputting the aggregation results corresponding to each global group set according to the sorting order of the grouping field values ​​to form the final aggregation query result corresponding to the consistency section identifier V.

[0019] This invention also provides a massive data sharding and aggregation query system based on a hybrid database architecture, including:

[0020] Query parsing module: Receives aggregate query requests, parses them to obtain the target table, filter conditions, grouping fields, aggregate fields and time range, and generates a consistency profile identifier V for the commit version at the time the aggregate query request is received;

[0021] Boundary compensation generation module: Under the constraint of consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and for the start boundary and end boundary of each basic segment, the boundary records within a preset range are extracted to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set;

[0022] Segment fingerprint generation module: Extracts each candidate segment in the candidate segment set in parallel and generates segment fingerprint information for each candidate segment;

[0023] Aggregate closure construction module: Based on the candidate shard number and shard fingerprint information, establish the mapping relationship between the basic shard and the bilateral compensation shard, and identify the associated records that cross the boundaries of adjacent basic shards and have the same grouping key based on the grouping field summary, and construct the corresponding aggregate closure unit;

[0024] Deduplication attribution rule generation module: Determines the target shard set participating in this aggregation query based on the aggregation closure unit, and generates deduplication attribution rules for each aggregation closure unit;

[0025] The result correction module performs local aggregation calculations on the target fragment set to obtain fragment local results. Based on the deduplication rules and fragment fingerprint information, it removes duplicate contribution values ​​in the fragment local results and compensates for boundary truncation contribution values ​​to obtain a complete correction result set.

[0026] Global aggregation module: Performs global aggregation on the correction result set according to the grouping field and aggregation field, and outputs the final aggregated query result corresponding to the consistency section identifier V.

[0027] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0028] 1. This invention constrains the entire query process based on a consistent cross-section identifier V, ensuring that multi-shard data extraction, parallel processing, and cross-engine aggregation are all based on a unified data version. This effectively avoids cross-shard data inconsistency caused by continuous data writing, fundamentally solving the technical defects of duplicate statistics, omissions, and result drift in aggregation results under massive data scenarios. Simultaneously, by introducing bilaterally compensated shards, shard fingerprint information, and aggregation closure units, data across shard boundaries can be accurately identified and completely recovered, achieving refined characterization and reconstruction of shard boundary semantics. This significantly improves the accuracy and stability of complex aggregation queries in high-concurrency environments.

[0029] 2. This invention, by constructing sharding mapping relationships, deduplication attribution rules, and a compensation mechanism based on boundary features, achieves precise removal of duplicate contribution values ​​and effective compensation for truncated data at boundaries, while ensuring query accuracy. This avoids the performance overhead and accuracy loss caused by traditional sharding aggregation relying on full merging or simple deduplication. Simultaneously, by combining parallel sharding extraction and local aggregation calculations, the computational burden is distributed from a single database node to multiple sharding processing units, significantly improving query performance and resource utilization efficiency without increasing the complexity of the system architecture. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 This is a flowchart of the method of the present invention.

[0032] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1, please refer to Figure 1 As shown in this embodiment, the massive data sharding and aggregation query method based on a hybrid database architecture includes:

[0035] S100: Receive the aggregation query request, parse the target table, filter conditions, grouping fields, aggregation fields and time range, and generate a consistency profile identifier V for the submission version at the time the aggregation query request is received.

[0036] In this invention, step S100 is used to establish a data visibility benchmark that is followed by all subsequent data processing procedures at the start of the aggregation query. Since the transactional database in a hybrid database architecture continuously receives new, modified, or reversed data, if the source data is directly sharded and extracted during the execution of the aggregation query, the shards extracted sequentially may correspond to data states at different submission times, leading to inconsistencies in the data foundation within the same query task. This ultimately results in duplicate statistics, omissions, or result drift in the aggregation results. Therefore, when receiving an aggregation query request, this invention first parses the query request and generates a consistency section identifier V based on the visibility range of the committed data in the transactional database at the time the request arrives, so that the subsequent data extraction, loading, and aggregation processing of each shard are all constrained by the same data version.

[0037] Specifically, after receiving an aggregation query request, the system first parses the query statement or query instruction, extracting the target table, filter conditions, grouping fields, aggregation fields, and time range corresponding to this query. The target table determines the data object containing the source data; the filter conditions limit the range of records participating in the calculation; the grouping fields determine the grouping keys for subsequent aggregations; the aggregation fields determine the objects of aggregation operations such as summation, counting, maximum, minimum, or average; and the time range limits the business time boundaries of the data. By extracting these parameters, a structured query description information for this aggregation query can be formed.

[0038] After obtaining the structured query description information, the system further accesses the transaction management information, version management information, or log sequence information of the transactional database to read the maximum visible commit version corresponding to the moment the aggregate query request is received. The maximum visible commit version refers to the upper bound of data versions that have been committed and are visible to the query task before that moment. This version can be a transaction commit number, log sequence number, timestamp version number, or other version identifier that can characterize the order of data commits. By obtaining this maximum visible commit version, "which data is allowed to be seen in this query" can be explicitly limited within a unified commit boundary.

[0039] Subsequently, the system associates and encapsulates the maximum visible committed version with the target table identifier, filtering conditions, grouping fields, aggregation fields, and time range to generate a consistency cross-section identifier V that corresponds one-to-one with this aggregation query. The consistency cross-section identifier V can be configured to include information such as the query task number, source table identifier, version upper bound, time interval, and query predicate summary. Its function is to provide unified version constraints for subsequent sharding, candidate shard extraction, boundary compensation shard generation, analytical database loading, and local aggregation calculations. In other words, all data records participating in this aggregation query should meet the following conditions: firstly, they must conform to the filtering conditions and time range; secondly, their committed version must not exceed the version upper bound defined by the consistency cross-section identifier V.

[0040] In some implementations, after generating a consistency section identifier V, the system writes this identifier V into the execution context of the current query task and passes it to subsequent processing modules. When determining shard boundaries, the sharding module sorts records and selects boundaries based on the visible data range defined by the consistency section identifier V. When extracting candidate shards in parallel, the data extraction module filters out data submitted after the upper bound of the current version based on the consistency section identifier V. When loading sharded data, the analytical database loading module only loads a subset of data that satisfies the consistency section constraint. When performing local aggregation and global merging, the aggregation calculation module uses the consistency section identifier V as a common prerequisite for the validity of the results. This ensures that even if new commit records are continuously generated in the transactional database during query execution, these new commit records will not be mixed into the current query task, thus ensuring that all sharded data originates from the same logical point-in-time data section.

[0041] For example, in a general ledger detail data query scenario, when a user initiates a request to summarize amounts grouped by accounting entity within a certain accounting period, if new accounting vouchers are still being written to the transactional database during the query execution period, the shards extracted at different times may contain different numbers of voucher details, leading to unstable final summary results. This invention addresses this by first obtaining the maximum visible version corresponding to the committed transaction at that moment when the request arrives and generating a consistency section identifier V. This ensures that all subsequent shards only extract data with a committed version no greater than the maximum visible version, thus making the final result correspond to the data state at a single, specific point in time. This not only avoids the problem of inconsistent statistical standards across shards but also provides a unified data foundation for subsequent disambiguation correction based on boundary-compensated sharding and aggregation closure units.

[0042] S200. Under the constraint of the consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and the boundary records of the preset range are extracted for the start boundary and end boundary of each basic segment to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set.

[0043] In this invention, step S200 is used to stably and orderly partition the target data that meets the query conditions under the constraint of the consistency section identifier V, and to solve the problem of incomplete aggregation semantics caused by data truncation at the partition boundary by introducing a boundary compensation mechanism. Since massive data is inevitably partitioned by primary key or time dimension, data records with the same grouping key will inevitably be distributed on both sides of the boundary of adjacent partitions. If aggregation calculations are directly performed on each partition separately, data in the same group will be split, resulting in duplicate counting or statistical omissions in the subsequent aggregation stage. Therefore, this invention further constructs double-sided compensation partitions in addition to the basic partitions to ensure data integrity across partition boundaries.

[0044] Specifically, within the data visibility range defined by the consistency section identifier V, the system first filters the data records in the transactional database based on the target table, filtering conditions, and time range parsed in step S100, obtaining a set of candidate records that meet the query conditions. On this basis, the candidate record set is globally sorted according to the primary key order of the target table, making the data logically form a monotonically ordered data sequence. The primary key can be a single field or a composite key, and its sorting order uniquely determines the sequential position of the records, thus providing a foundation for subsequent stable sharding.

[0045] Subsequently, the system sequentially segments the ordered data sequence based on a preset sharding strategy to obtain multiple basic shards. The sharding strategy can be configured based on a record count threshold, a data size threshold, or a time interval threshold, ensuring a relatively balanced data volume across the basic shards, thereby improving the efficiency of subsequent parallel processing. Each basic shard corresponds to a continuous primary key interval, with its starting boundary determined by the minimum primary key value in the shard and its ending boundary determined by the maximum primary key value in the shard.

[0046] After obtaining the basic fragments, to avoid truncation of grouped data at fragment boundaries, the system further constructs bilaterally compensated fragments for each basic fragment. Specifically, for any basic fragment, the system extracts boundary records within a preset range forward and backward from the ordered data sequence, centered on the start and end boundaries of that basic fragment. The preset range can be a fixed number of records, a fixed time span, or a dynamic range obtained based on changes in grouping fields, used to cover the same grouped data that may cross fragment boundaries. By extracting boundary records from both sides of the basic fragment, a front-side compensated fragment and a back-side compensated fragment corresponding to that basic fragment are formed, thus constituting a bilaterally compensated fragment structure.

[0047] Furthermore, the system combines the base fragment with its corresponding bilateral compensation fragment to form a candidate fragment set. Each candidate fragment in the candidate fragment set contains not only its main data but also additional data used to compensate for fragment boundaries, ensuring that any candidate fragment can cover the entire range of potentially cross-fragment boundary grouped data when performing local aggregation calculations. It should be noted that some records in the compensation fragment may be duplicated in adjacent candidate fragments, but this duplication provides the necessary data redundancy basis for subsequent disambiguation processing based on fragment fingerprint information and deduplication attribution rules.

[0048] In some implementations, to improve the stability and reproducibility of fragmentation, the system uses the data version defined by the consistency section identifier V as the boundary when generating basic fragments and bilateral compensation fragments, ensuring that the fragmentation process is not affected by subsequent additions or modifications to the data. Simultaneously, the system can also generate fragment identifiers and boundary descriptions for each basic fragment and its corresponding bilateral compensation fragment, for use in subsequent fragment loading, fragment mapping, and aggregation closure unit construction.

[0049] For example, when partitioning general ledger detail data according to primary key order, if multiple voucher details of a certain legal entity happen to be distributed at the boundary of two adjacent basic partitions, then by introducing compensation partitions on both sides of the boundary, this part of cross-boundary data of the same legal entity can appear simultaneously in adjacent candidate partitions. Thus, when processing in conjunction with deduplication rules, the complete aggregation semantics of the legal entity under a single group can be restored, avoiding statistical distortion caused by boundary partitioning.

[0050] S300. Extract each candidate segment from the candidate segment set in parallel and generate segment fingerprint information for each candidate segment.

[0051] In this invention, step S300 is used to efficiently extract each candidate fragment in parallel after the candidate fragment set is generated, and to construct fragment fingerprint information that can characterize the fragment data structure features and boundary semantics, thereby providing a basis for subsequent fragment mapping, cross-fragment data disambiguation, and construction of aggregation closure units. Since candidate fragments include basic fragments and their bilateral compensation fragments, there is some data overlap and boundary intersection. Without an effective data feature identification mechanism, it will be difficult to identify duplicate contributions and boundary truncation sources in the subsequent aggregation process. Therefore, this invention generates fragment fingerprint information for each candidate fragment, enabling each fragment to have a comparable, verifiable, and traceable structured identifier.

[0052] Specifically, under the constraint of the consistency section identifier V, the system initiates parallel extraction tasks for each candidate shard in the candidate shard set. These parallel extraction tasks can be implemented based on a multi-threaded or distributed execution framework. Each candidate shard corresponds to an independent extraction unit, which reads data records from the transactional database according to the shard boundary range and outputs them as structured data files or data streams. During the extraction process, each candidate shard is strictly constrained by the consistency section identifier V, and only data records whose submission version does not exceed the consistency section identifier V are extracted, thereby ensuring that the data of all candidate shards originates from the same logical point in time.

[0053] While extracting data, the system generates corresponding shard fingerprint information for each candidate shard. This shard fingerprint information describes the data boundary characteristics, content distribution characteristics, and grouping semantic characteristics of the candidate shard, and includes at least the following: shard number, shard type identifier (basic shard or compensation shard), minimum primary key value, maximum primary key value, total number of records, boundary record summary, and grouping field distribution summary. The boundary record summary characterizes the features of records near the shard's start and end boundaries, and can be obtained by hashing or feature encoding the primary key and grouping fields of the boundary records. The grouping field distribution summary describes the distribution of grouping keys within the shard, and can be represented using a grouping key hash set, frequency statistics vector, or compressed Bloom structure.

[0054] Furthermore, in some implementations, the system can also generate a global hash check value based on all or part of the data records within a candidate fragment, which can be used to quickly determine whether there is data overlap or data consistency relationship between different candidate fragments. For bilateral compensation fragments, their fragment fingerprint information can also be additionally marked with their source base fragment number and its corresponding boundary direction (front or back) so as to identify their compensation semantics in subsequent processing.

[0055] The fragment fingerprint information generated in the above manner corresponds one-to-one with the candidate fragment data and is transmitted to the analytical database along with the data. In subsequent steps, the system can match the boundary relationships between different candidate fragments based on the fragment fingerprint information. For example, it can identify adjacent fragments by comparing primary key ranges and boundary record summaries, and identify potential cross-fragment data of the same group by comparing grouping field distribution summaries, thus providing a basis for constructing aggregation closure units. Simultaneously, when performing disambiguation of local aggregation results, the fragment fingerprint information can be used to determine the source fragment of duplicate records and their attribution, achieving accurate removal of duplicate contribution values ​​and compensation and recovery of truncated boundary data.

[0056] For example, in the general ledger detail scenario, if a certain accounting subject has multiple consecutive records distributed across two candidate shards at the shard boundary, by comparing the grouping field distribution summary and boundary record summary of the two candidate shards, it can be identified that the data corresponding to the accounting subject belongs to a continuous distribution across shards. Thus, it can be included in the same aggregation closure unit for processing in the subsequent aggregation process, and will not be misjudged as two independent groups.

[0057] S400. Establish the mapping relationship between the basic fragment and the bilateral compensation fragment based on the fragment number and fragment fingerprint information of each candidate fragment, and identify the associated records that cross the boundaries of adjacent basic fragments and have the same grouping key based on the grouping field summary, and construct the corresponding aggregation closure unit.

[0058] In this embodiment, boundary semantic similarity is used to characterize the degree of data continuity and semantic connection between two candidate shards at the primary key boundary. This boundary semantic similarity is not directly determined based on the candidate shard number, but rather calculated based on the shard fingerprint information of the candidate shards. Specifically, the records corresponding to each candidate shard are first stably sorted according to the primary key from smallest to largest. Then, the first 32 records and the last 32 records after sorting are extracted as boundary records. For each boundary record, the primary key field value, grouping field value, and time field value are extracted sequentially. Numerical fields are uniformly converted to decimal characters, time fields are uniformly converted to the format "year-month-day hour:minute:second," and grouping fields are converted to a fixed-length character sequence according to a preset dictionary order. These are then concatenated in the order of "primary key field value - grouping field value - time field value" to form a record description string. For each record description string, a 256-bit secure hash algorithm is used to generate a digest value, which is then concatenated according to the original record order to form a boundary record digest, thus obtaining the starting boundary record digest corresponding to the candidate shard's starting boundary and the ending boundary record digest corresponding to the ending boundary.

[0059] After obtaining the boundary record summaries, the interval adjacency value, boundary summary match value, and boundary order continuation value are calculated for any two candidate fragments. The interval adjacency value is calculated as follows: first, determine whether the maximum primary key value of one candidate fragment is adjacent to or overlaps with the minimum primary key value of another candidate fragment. When the difference between their primary keys is between 0 and 128, a linear score is assigned based on the principle that the smaller the difference, the higher the interval adjacency value. When the primary key intervals overlap, a score is assigned based on the proportion of overlapping records to the total number of boundary records. The boundary summary match value is calculated as follows: the end boundary record summary of the previous candidate fragment is compared with the start boundary record summary of the next candidate fragment in record order. The number of records with completely identical summary values ​​is counted, and then this number is divided by 32 to obtain the boundary summary match value between 0 and 1. The boundary order continuation value is calculated as follows: The record description strings corresponding to the two sets of boundary record summaries are parsed in reverse. It is determined whether the grouping field values ​​appear consecutively and whether the time field values ​​are kept in non-reverse order. The more record pairs that meet this condition (consecutive appearance and non-reverse order), the higher the boundary order continuation value. Finally, the boundary semantic similarity between the two candidate segments is obtained by weighting the interval adjacency values ​​(40 points), boundary summary matching values ​​(35 points), and boundary order continuation values ​​(25 points). The result is rounded to three decimal places.

[0060] In this embodiment, the sharding topology graph is used to represent the boundary connection status and possible compensation affiliation relationships between candidate shards. The sharding topology graph is constructed using a combination of a node table and an edge table. The node table records each candidate shard number and its corresponding minimum primary key value, maximum primary key value, start boundary record summary, and end boundary record summary. The edge table records whether a connection is established between two candidate shards and the corresponding boundary semantic similarity. During construction, all candidate shards are first traversed in natural order of their shard numbers, and each candidate shard number is registered as an independent node. Then, boundary semantic similarity calculation is performed on any two candidate shards. When two candidate shards meet any one of the following conditions: "adjacent primary key intervals, overlapping primary key intervals, or primary key interval difference not exceeding 128", they are listed as a pair of nodes to be connected.

[0061] For each pair of nodes to be connected, the obtained boundary semantic similarity is compared with a preset connection threshold. In this embodiment, the preset connection threshold is set to 0.55. When the boundary semantic similarity is greater than or equal to 0.55, a connection record is written into the edge relationship table. The connection record includes at least the starting candidate fragment number, the target candidate fragment number, the boundary direction, the boundary semantic similarity, and the primary key interval relationship. The boundary direction is determined based on the minimum and maximum primary key values ​​of the two candidate fragments: if the maximum primary key value of the first candidate fragment is less than or equal to the minimum primary key value of the second candidate fragment, the boundary direction is recorded as "forward connection"; if the minimum primary key value of the first candidate fragment is greater than or equal to the maximum primary key value of the second candidate fragment, the boundary direction is recorded as "backward connection"; if there is an overlap in the primary key intervals, the boundary direction is recorded as "overlapping connection". The fragment topology graph constructed in this way is essentially a weighted directed graph, which retains both the sequential relationship between candidate fragment numbers and the boundary continuity relationship expressed by the boundary semantic similarity, providing a data foundation for subsequent identification of the mapping relationship between the basic fragments and the bilateral compensation fragments.

[0062] In this implementation, the mapping between the base shard and the bilateral compensation shard does not indiscriminately retain all connected nodes, but rather performs targeted filtering with the base shard as the central node. Specifically, the process first locates the candidate shard number corresponding to a given base shard in the shard topology graph, and determines this candidate shard number as the central node. Then, it reads all candidate shard numbers directly connected to this central node from the edge relationship table and categorizes them into forward connection nodes, backward connection nodes, and overlapping connection nodes according to the boundary direction. For a forward connection node, it indicates that it is located after the primary key interval of the base shard; for a backward connection node, it indicates that it is located before the primary key interval of the base shard; and for an overlapping connection node, it indicates that it intersects with the base shard at the boundary position.

[0063] After direction segmentation, each connected node is individually assigned a compensation attribute. In this embodiment, the preset threshold is set to 0.75. Only connected nodes that simultaneously meet the conditions of "adjacent or overlapping primary key intervals" and "boundary semantic similarity greater than 0.75" are considered to have a compensation relationship with the base segment. Furthermore, to avoid confusion in subsequent assignments caused by one base segment corresponding to multiple bilateral compensation segments in the same direction, multiple connected nodes in the same direction are sorted from high to low according to boundary semantic similarity, and the candidate segment number ranked first is selected as the bilateral compensation segment node in that direction; when the difference in boundary semantic similarity between the first and second ranked nodes is less than 0.05, the difference in primary key intervals between the two candidate segments and the base segment is compared, and the one with the smaller primary key interval difference is selected first. Through the above processing, each base segment ultimately obtains at most one front bilateral compensation segment node and one back bilateral compensation segment node, and the selection result is unique and repeatable.

[0064] In this implementation, the mapping relationships are stored in the form of mapping records. Each mapping record includes at least the basic fragment number, the front-side bilateral compensation fragment number, the back-side bilateral compensation fragment number, the forward boundary semantic similarity, the backward boundary semantic similarity, and the boundary direction identifier. When no bilateral compensation fragment node meeting the conditions is found in a certain direction, the bilateral compensation fragment number corresponding to that direction is recorded as "0", indicating that there is no bilateral compensation fragment in that direction. After the mapping records are established, a mapping relationship table is formed in ascending order of the basic fragment numbers, so that each basic fragment has a fixed compensation attribution description.

[0065] The aforementioned mapping table is directly used to construct subsequent aggregation closure units. Specifically, the mapping record corresponding to a given base shard number is first read, and then all boundary record summaries of that base shard and its preceding and following double-sided compensation shards are extracted. Subsequently, the grouping field values ​​in the boundary record summaries of these three shards are parsed one by one, and records with the same grouping field values ​​and located within adjacent primary key intervals are grouped into the same candidate aggregation set. Based on the candidate aggregation set, integrity is verified according to three conditions: continuous primary key intervals, consistent grouping field values, and non-reverse order time field values. Candidate aggregation sets that meet these three conditions are determined as an aggregation closure unit. In other words, the aggregation closure unit is not solely composed of records within the base shard, but rather consists of related records from the base shard and the double-sided compensation shards identified through the mapping relationship. After this processing, the same grouped data, originally truncated by shard boundaries, can be restored to a continuous data range within the aggregation closure unit, thus providing a complete and consistent data foundation for subsequent local aggregation and global summarization.

[0066] In this embodiment, the grouping field summary is used to characterize the distribution characteristics of each grouping key within a candidate segment, and the boundary record summary is used to characterize the structural characteristics of records at the boundaries of the candidate segment. Specifically, for adjacent basic segments and their corresponding bilateral compensation segments, the grouping field summary and boundary record summary are read from their segment fingerprint information, respectively. Then, the boundary record summary is reverse-parsed, and according to a pre-agreed encoding rule, the summary value is restored to the corresponding record description string sequence. Then, the grouping field value, primary key field value, and time field value are extracted sequentially from the record description string. After extraction, all boundary records are sorted in ascending order of primary key field value and identified by the grouping field value to construct a grouping key distribution sequence. The grouping key distribution sequence is represented in the form of an ordered list, where each element includes at least a grouping field value, a corresponding primary key field value, and a time field value, thus providing basic data for subsequent cross-segment continuity identification.

[0067] In this embodiment, candidate record segment identification is accomplished through sequential scanning. Specifically, the grouping key distribution sequence corresponding to the end boundary of adjacent basic segments is concatenated with the grouping key distribution sequence corresponding to the start boundary of the subsequent two-sided compensation segments to form a continuous sequence. Then, starting from the first element of this continuous sequence, the grouping field values ​​are compared one by one. When at least three consecutive records with the same grouping field value appear, the continuous interval is marked as a candidate record segment. For each candidate record segment, a primary key continuity condition is further performed, i.e., it is determined whether the primary key difference between adjacent records is not greater than 1. If there is an interval greater than 1, the candidate record segment is truncated. At the same time, a time order consistency condition is performed, i.e., it is determined whether the time field values ​​of the records maintain a non-decreasing order. If there is a reverse order, the candidate record segment is also truncated. Finally, the candidate record segments that simultaneously satisfy the primary key continuity condition and the time order consistency condition are determined as the associated record set.

[0068] In this embodiment, cross-shard grouping is based on the associated record set. Specifically, each associated record set is first grouped according to the grouping field value, with records having the same grouping field value grouped into the same subset. Then, the records in each subset are sorted according to the primary key field value, and their primary key intervals are checked for continuity. If a discontinuity exists, the subset is split into multiple sub-intervals based on the discontinuity position. Next, within each sub-interval, the time field value is sorted, and it is verified whether the time field values ​​constitute a continuous time interval, i.e., the time difference between adjacent records does not exceed a preset time window, which is set to 60 seconds in this embodiment. When both the primary key interval and the time interval are continuous, the sub-interval is determined as a closure candidate range. Each closure candidate range includes at least the grouping field value, the starting primary key field value, the ending primary key field value, the starting time field value, and the ending time field value.

[0069] In this embodiment, integrity verification is used to ensure that the candidate closure range can completely cover the same grouped data across shard boundaries. The specific verification process includes three parts: First, verifying the number of records within the candidate closure range by comparing the number of records with the statistical count of the grouped field value in the corresponding base shard and the bilateral compensation shard; a difference of 0 indicates a consistency. Second, performing consistency verification on the boundary records within the candidate closure range, i.e., comparing whether the starting position of the candidate closure range is consistent with the corresponding record in the record summary of the previous shard boundary, and simultaneously comparing whether the ending position is consistent with the corresponding record in the record summary of the next shard boundary. Third, performing uniqueness verification on the grouped field values ​​within the candidate closure range to ensure that no records with different grouped field values ​​are mixed in within this range. Only when all three verifications pass is the candidate closure range determined as an aggregated closure unit. The aggregated closure unit serves as the smallest complete data unit for subsequent local aggregation calculations and global result merging, used to eliminate the truncation effect caused by shard boundaries on the aggregation results.

[0070] S500. Determine the target shard set to participate in this aggregation query based on the aggregation closure unit, and generate deduplication rules for each aggregation closure unit.

[0071] In this implementation, each aggregation closure unit is first traversed, and the grouping field value, starting primary key field value, ending primary key field value, and the base shard number and bilateral compensation shard number associated with that aggregation closure unit are read. For each aggregation closure unit, its associated base shard number and bilateral compensation shard number are sequentially written into the candidate shard set. After traversing all aggregation closure units, the shard numbers in the candidate shard set are sorted according to their numerical values, and duplicate shard numbers are removed using a sequential scanning method, thereby forming the target shard set. The target shard set is used to limit the range of data sources participating in subsequent aggregation calculations, ensuring that only data shards associated with the aggregation closure units are processed.

[0072] In this embodiment, for each record within each aggregation closure unit, the position of the record in the base shard and the bilateral compensation shard is located. Specifically, the primary key field value of the record is compared with the minimum and maximum primary key values ​​of each shard to determine the relative position of the record in the corresponding shard, and its distance from the shard boundary is calculated. The calculation of the attribution priority includes two dimensions: one is the primary key position offset, which is the distance between the primary key field value of the record and the center position of the primary key interval of the shard; the smaller the distance, the higher the priority. The second is the boundary distance, which is the minimum distance between the record and the start or end boundary of the shard; the larger the distance, the higher the priority. These two dimensions are comprehensively evaluated according to a weighting of 60 parts for the primary key position offset and 40 parts for the boundary distance to obtain the attribution priority score of the record in each shard. The score range is limited to 0 to 1 and retained to 3 decimal places.

[0073] For each record, its priority score in the base shard and the two-sided compensation shards is compared, and the shard with the highest score is selected as the unique shard to which the record belongs. When two or more shards have the same priority score and it is the highest value, the boundary distances of the record in each shard are further compared, and the shard with the larger boundary distance is selected first; if the boundary distances are still the same, the shard with the smaller primary key interval length is selected as the unique shard to which the record belongs. After the unique shards to which the records belong are assigned, the unselected shards are marked as redundant shards. In this way, each record belongs to only one shard, thus providing a clear basis for subsequent elimination of duplicate contributions.

[0074] In this embodiment, the deduplication attribution rules are constructed based on record-level attribution results. Specifically, for each aggregation closure unit, records are grouped according to the grouping field value, and within each group, the set of records assigned to a unique attribution shard in each shard is counted. For the same grouping field value, only records in the uniquely attribution shard are retained for subsequent aggregation calculations; corresponding records in redundant shards are not included in the aggregation calculation. Furthermore, to improve execution efficiency, the deduplication attribution rules are transformed into a set of executable filtering conditions. Each filtering condition includes at least the shard number, the primary key field value range, and the grouping field value range. During subsequent local aggregation calculations, only records satisfying the filtering conditions are counted, thereby eliminating duplicate contributions and ensuring unified attribution across shards, guaranteeing the uniqueness and consistency of the final aggregation result.

[0075] S600. Perform local aggregation calculation on the target fragment set to obtain fragment local results. Based on the deduplication attribution rules and fragment fingerprint information, remove duplicate contribution values ​​in the fragment local results and compensate for boundary truncation contribution values ​​to obtain a complete closure correction result set.

[0076] In this embodiment, each target fragment in the target fragment set is first processed separately. Specifically, data records satisfying the consistency section identifier V constraint are read from each target fragment, and the records are grouped according to the grouping field. Records with the same grouping field value are grouped into the same group set. Within each group set, the corresponding aggregation calculation is performed according to the type of the aggregation field: when the aggregation field is a numeric field, the values ​​of this field for all records in the group are summed one by one to obtain the sum; when the aggregation field is a count field, the number of records is counted; when the aggregation field is a maximum or minimum value field, the records are compared one by one to obtain the maximum or minimum value. After the calculation is completed, the grouping field value corresponding to each group set is combined with the aggregation result value to form a fragmented local result, and recorded in the structure "fragment number - grouping field value - aggregation result value".

[0077] In this implementation, the deduplication attribution rules are executed based on record-level attribution relationships. Specifically, the original record set corresponding to each shard's local result is first located according to the filtering conditions defined in the deduplication attribution rules. For each record, it is determined whether its shard is a uniquely attributed shard: if the shard number of the record matches the uniquely attributed shard number specified in the deduplication attribution rules, the record's contribution value is retained; if they do not match, the record is marked as a duplicate contribution record and does not participate in subsequent calculations. For results that have already undergone local aggregation, a reverse mapping process is required, that is, the aggregated values ​​in the shard's local result are split according to the original number of records to restore the record-level contribution values, and then filtered according to the above rules. After filtering, the aggregation calculation is re-executed according to the grouping field to obtain the preliminary deduplication result after removing duplicate contribution values, thereby ensuring that each record is counted only once in the uniquely attributed shard.

[0078] In this embodiment, the identification of boundary truncation contributions is achieved by comparing the boundary features of the base shard and the bilateral compensation shard. Specifically, the minimum and maximum primary key values, as well as the boundary record summary, are first read from the shard fingerprint information. The boundary record summary is then reverse-parsed to obtain the grouping field value sequence of the boundary records. Subsequently, the ending boundary record of the base shard is compared line by line with the starting boundary record of its subsequent bilateral compensation shard. When a record sequence with identical grouping field values ​​but not fully covered in the base shard is found, the primary key interval corresponding to that sequence is marked as a missing contribution interval. For each missing contribution interval, all records meeting the conditions are extracted from the corresponding bilateral compensation shard as compensation records based on their primary key range. To avoid overcompensation, a range check is performed on the compensation records, requiring their primary key field values ​​to be within the missing contribution interval range and their time field values ​​to meet the time continuity condition with the corresponding records in the base shard. The time continuity condition is that the time difference between adjacent records does not exceed 60 seconds.

[0079] In this implementation, the generation of the correction result is divided into two stages: compensation merging and final aggregation. First, the preliminary deduplication result is merged with the compensation records. For each grouping field value, the aggregation field value in the compensation record is added to the corresponding preliminary deduplication result according to the same grouping field value, thereby restoring the contribution value missing due to boundary truncation. Then, the merged result is aggregated again according to the grouping fields to ensure that all records belonging to the same grouping field value are counted uniformly. To ensure the integrity of the result, a consistency check is performed on the final result, comparing the number of records corresponding to each grouping field value with the number of records in the aggregation closure unit. When they match, the correction is considered complete. Finally, all grouping field values ​​and their corresponding aggregation result values ​​are combined to form a complete correction result set for subsequent global aggregation processing.

[0080] S700. Perform global aggregation on the correction result set according to the grouping field and aggregation field, and output the final aggregation query result corresponding to the consistency section identifier V.

[0081] In this implementation, all records in the correction result set are first sorted according to their grouping field values. Specifically, the grouping field values ​​are uniformly converted into fixed-length character sequences, and then compared and sorted lexicographically to obtain an ordered sequence of records. After sorting, a sequential scanning method is used to divide the records. When adjacent records have the same grouping field value, they are grouped into the same global group set; when the grouping field value changes, the current global group set is closed and a new global group set is started. Each global group set contains at least multiple records with identical grouping field values ​​and retains the corresponding aggregation field value and source identifier for subsequent global aggregation calculations.

[0082] For each global group set, a second aggregation calculation is performed. Specifically, the aggregation method is first determined based on the aggregation field type: when the aggregation field is a numerical accumulation type, the aggregation field values ​​of all records in the set are accumulated one by one; when the aggregation field is a count type, the number of records is counted; when the aggregation field is an extreme value type, all records in the set are compared one by one to obtain the maximum or minimum value. After completing the basic aggregation calculation, a result constraint rule based on the consistency section identifier V is introduced to filter the records participating in the calculation. The implementation of the result constraint rule is as follows: the time field value and version identifier of each record are verified, and only records whose time field value is within the time range corresponding to the consistency section identifier V and whose submitted version does not exceed the upper bound of the version limit defined by the consistency section identifier V are retained. For records that do not meet the above conditions, their aggregation contribution value is removed from the aggregation result of the current global group set, and the aggregation result is recalculated, thereby ensuring that the global result strictly corresponds to the consistency section identifier V.

[0083] In this embodiment, to ensure the traceability and uniqueness of the final results, a unique result identifier is generated for each global group set. Specifically, the grouping field value, aggregation field value, and consistency section identifier V corresponding to the global group set are concatenated and a string sequence is generated in a fixed order. This string sequence is then subjected to a 128-bit hash calculation to obtain the unique result identifier. Subsequently, the unique result identifier is associated with the corresponding grouping field value and aggregation field value to construct a global result mapping relationship. This global result mapping relationship is stored in key-value pair format, where the key is the unique result identifier, and the value is the grouping field value and its corresponding aggregation result value, thereby achieving unique identification and fast retrieval for each group result.

[0084] After constructing the global result mapping relationship, the aggregation results corresponding to all global group sets are output uniformly. Specifically, the results in the global result mapping relationship are first sorted according to the order of the grouping field values. Then, the sorted result items are read sequentially, and the grouping field value and aggregation field value in each result are combined according to a preset output format. This output format includes at least the grouping field value, the aggregation result value, and the corresponding consistency section identifier V, indicating the time consistency range of the result. Finally, all results are output sequentially, forming the final aggregation query result that strictly corresponds to the consistency section identifier V, thus completing the closed-loop processing of the entire massive data sharding aggregation query process.

[0085] Example 2, please refer to Figure 2 As shown in this embodiment, the massive data sharding aggregation query system based on a hybrid database architecture includes:

[0086] Query parsing module: Receives aggregate query requests, parses them to obtain the target table, filter conditions, grouping fields, aggregate fields and time range, and generates a consistency profile identifier V for the commit version at the time the aggregate query request is received;

[0087] Boundary compensation generation module: Under the constraint of consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and for the start boundary and end boundary of each basic segment, the boundary records within a preset range are extracted to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set;

[0088] Segment fingerprint generation module: Extracts each candidate segment in the candidate segment set in parallel and generates segment fingerprint information for each candidate segment;

[0089] Aggregate closure construction module: Based on the candidate shard number and shard fingerprint information, establish the mapping relationship between the basic shard and the bilateral compensation shard, and identify the associated records that cross the boundaries of adjacent basic shards and have the same grouping key based on the grouping field summary, and construct the corresponding aggregate closure unit;

[0090] Deduplication attribution rule generation module: Determines the target shard set participating in this aggregation query based on the aggregation closure unit, and generates deduplication attribution rules for each aggregation closure unit;

[0091] The result correction module performs local aggregation calculations on the target fragment set to obtain fragment local results. Based on the deduplication rules and fragment fingerprint information, it removes duplicate contribution values ​​in the fragment local results and compensates for boundary truncation contribution values ​​to obtain a complete correction result set.

[0092] Global aggregation module: Performs global aggregation on the correction result set according to the grouping field and aggregation field, and outputs the final aggregated query result corresponding to the consistency section identifier V.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for massive data sharding and aggregation query based on a hybrid database architecture, characterized by: include: S100: Receive an aggregation query request, parse the target table, filter conditions, grouping fields, aggregation fields and time range, and generate a consistency profile identifier V for the submission version at the time the aggregation query request is received; S200. Under the constraint of consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and the boundary records of the preset range are extracted for the start boundary and end boundary of each basic segment to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set. S300: Extract each candidate segment in the candidate segment set in parallel and generate segment fingerprint information for each candidate segment. S400. Establish the mapping relationship between the basic fragment and the bilateral compensation fragment based on the fragment number and fragment fingerprint information of each candidate fragment, and identify the associated records that cross the boundaries of adjacent basic fragments and have the same grouping key based on the grouping field summary, and construct the corresponding aggregation closure unit. S500. Determine the target shard set to participate in this aggregation query based on the aggregation closure unit, and generate deduplication rules for each aggregation closure unit. S600. Perform local aggregation calculation on the target fragment set to obtain fragment local results. Based on the deduplication and fragment fingerprint information, remove duplicate contribution values ​​in the fragment local results and compensate for boundary truncation contribution values ​​to obtain a complete closure correction result set. S700. Perform global aggregation on the correction result set according to the grouping field and aggregation field, and output the final aggregation query result corresponding to the consistency section identifier V.

2. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 1, characterized in that: The step of establishing a mapping relationship between the basic shard and the bilateral compensation shard based on the candidate shard number and shard fingerprint information includes: using the candidate shard number as a node, constructing a shard topology graph based on boundary semantic similarity; when two candidate shards satisfy the conditions of adjacent primary key intervals, overlapping primary key intervals, or a primary key interval difference not exceeding 128, listing them as node pairs to be connected; when the boundary semantic similarity of the node pairs to be connected is greater than or equal to 0.55, establishing a connection relationship in the shard topology graph, and determining the boundary direction based on the minimum and maximum primary key values; using the candidate shard number corresponding to the basic shard as the center node, selecting connection nodes with a boundary semantic similarity greater than 0.75 as bilateral compensation shard nodes, and determining the front bilateral compensation shard and the back bilateral compensation shard according to the boundary semantic similarity from high to low, and establishing a mapping relationship table.

3. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 2, characterized in that: The construction of the corresponding aggregation closure unit specifically includes: extracting the grouping field summary and boundary record summary of adjacent basic segments and their corresponding bilateral compensation segments, and restoring the grouping key distribution sequence in the boundary records according to the primary key order; concatenating the grouping key distribution sequence corresponding to the end boundary of the adjacent basic segment with the grouping key distribution sequence corresponding to the start boundary of the subsequent bilateral compensation segment, and determining the continuous interval as a candidate record segment when at least 3 consecutive records with the same grouping field value appear; performing primary key continuity condition and time order consistency condition judgment on the candidate record segment, and determining the candidate record segment that satisfies the primary key difference not greater than 1 and the time field value maintains a non-decreasing order as the associated record set; merging the associated record set according to the same grouping field value, continuous primary key interval and continuous time interval, and determining the aggregation closure unit after checking the record quantity consistency, boundary record consistency and grouping field value uniqueness.

4. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 1, characterized in that: S100 specifically includes: parsing the aggregation query request, extracting the target table, filtering conditions, grouping fields, aggregation fields, and time range; reading the maximum visible commit version corresponding to the moment the aggregation query request is received; associating and encapsulating the maximum visible commit version with the target table identifier, filtering conditions, grouping fields, aggregation fields, and time range to generate a consistency profile identifier V.

5. The massive data sharding and aggregation query method based on a hybrid database architecture according to claim 1, characterized in that: The fragment fingerprint information generated in S300 includes: fragment number, fragment type identifier, minimum primary key value, maximum primary key value, total number of records, boundary record summary, and grouping field summary; wherein, the boundary record summary is generated by encoding the primary key field value, grouping field value, and time field value of records near the start and end boundaries of the candidate fragment, and the grouping field summary is used to describe the distribution of each grouping key within the candidate fragment; for bilaterally compensated fragments, the fragment fingerprint information also includes the source base fragment number and boundary direction.

6. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 1, characterized in that: The specific steps in generating deduplication attribution rules in S500 include: for each record within each aggregation closure unit, calculating the attribution priority score of the record in each shard based on the coverage range of the primary key field value in the base shard and the bilateral compensation shard; wherein, the attribution priority score is determined by a combination of the primary key position offset and the boundary distance, with the primary key position offset accounting for 60 parts and the boundary distance accounting for 40 parts; selecting the shard with the highest attribution priority score as the unique attribution shard for the record, and marking the remaining shards as redundant shards; generating a set of filtering conditions according to the shard number, the range of primary key field values, and the range of grouping field values ​​to form the deduplication attribution rules.

7. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 1, characterized in that: S600 specifically includes: grouping data records in each target fragment of the target fragment set according to the grouping field, and performing local aggregation calculation according to the type of the aggregation field to obtain fragment local results; according to the deduplication attribution rules, performing record-level filtering on the original record set corresponding to the fragment local results, retaining only the record contribution values ​​in the uniquely attributed fragments, and removing duplicate contribution values ​​in redundant fragments to obtain preliminary deduplication results; identifying missing contribution intervals based on the minimum primary key value, maximum primary key value, and boundary record summary in the fragment fingerprint information, and extracting compensation records that meet the primary key range and time continuity conditions from the corresponding bilateral compensation fragments; merging the preliminary deduplication results with the compensation records, and performing aggregation calculation again according to the grouping field to obtain a complete closure correction result set.

8. The method for massive data sharding and aggregation query based on a hybrid database architecture according to claim 1, characterized in that: S700 specifically includes: sorting all records in the correction result set according to the grouping field value, and dividing records with the same grouping field value into the same global group set; performing aggregation calculation again in each global group set according to the type of the aggregation field, and filtering records according to the time range and version upper bound corresponding to the consistency section identifier V; concatenating the grouping field value, aggregation field value and consistency section identifier V corresponding to each global group set to generate a unique result identifier, and constructing a global result mapping relationship; outputting the aggregation results corresponding to each global group set according to the sorting order of the grouping field values, forming the final aggregation query result corresponding to the consistency section identifier V.

9. A massive data sharding and aggregation query system based on a hybrid database architecture, used to implement the massive data sharding and aggregation query method based on a hybrid database architecture as described in any one of claims 1-8, characterized in that: include: Query parsing module: Receives aggregate query requests, parses them to obtain the target table, filter conditions, grouping fields, aggregate fields and time range, and generates a consistency profile identifier V for the commit version at the time the aggregate query request is received; Boundary compensation generation module: Under the constraint of consistency section identifier V, the records that meet the filtering conditions are sequentially divided according to the primary key order of the target table and the time range to obtain multiple basic segments; and for the start boundary and end boundary of each basic segment, the boundary records within a preset range are extracted to form a double-sided compensation segment corresponding to the basic segment, thereby obtaining a candidate segment set; Segment fingerprint generation module: Extracts each candidate segment in the candidate segment set in parallel and generates segment fingerprint information for each candidate segment; Aggregate closure construction module: Based on the candidate shard number and shard fingerprint information, establish the mapping relationship between the basic shard and the bilateral compensation shard, and identify the associated records that cross the boundaries of adjacent basic shards and have the same grouping key based on the grouping field summary, and construct the corresponding aggregate closure unit; Deduplication attribution rule generation module: Determines the target shard set participating in this aggregation query based on the aggregation closure unit, and generates deduplication attribution rules for each aggregation closure unit; The result correction module performs local aggregation calculations on the target fragment set to obtain fragment local results. Based on the deduplication rules and fragment fingerprint information, it removes duplicate contribution values ​​in the fragment local results and compensates for boundary truncation contribution values ​​to obtain a complete correction result set. Global aggregation module: Performs global aggregation on the correction result set according to the grouping field and aggregation field, and outputs the final aggregated query result corresponding to the consistency section identifier V.

Images