Data segmentation and query processing system under distributed architecture

Abstract

The invention relates to the technical field of computer data processing, and discloses a data segmentation and query processing system under a distributed architecture. The system comprises a data segmentation module which divides data blocks based on a self-adaptive partitioning algorithm; the distributed storage module is used for distributing data blocks by using a consistent Hash algorithm and dynamically adjusting the distribution of virtual nodes; the index construction module is used for generating a global index table through a multi-layer inverted index algorithm; the query optimization module is used for converting query statements by utilizing a query rewriting algorithm; and the result aggregation module is used for processing the sub-query results by adopting a parallel aggregation algorithm. The system can dynamically process data according to data characteristics, optimizes the data storage and query process, effectively improves the data processing efficiency and query performance of a distributed system, and is suitable for scenes such as big data storage and analysis.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a data segmentation and query processing system under a distributed architecture. Background Technology

[0002] With the rapid development of information technology, data volume is exploding, and traditional data processing and querying methods are gradually revealing numerous problems when faced with massive amounts of data. In terms of data storage, centralized storage systems are limited by the capacity and performance bottlenecks of individual storage devices, making it difficult to meet the ever-increasing data storage demands. When the data volume exceeds the carrying capacity of storage devices, not only will storage costs increase dramatically, but there is also a risk of data loss or corruption. For example, some small and medium-sized enterprises that initially adopted centralized databases often experience insufficient storage space as their business expands and data volume continues to climb. Frequent replacement of storage devices is not only costly but also disrupts normal business operations.

[0003] In data query processing, traditional single-node query methods are inefficient. For complex queries, a single node needs to traverse the entire dataset, which consumes significant time and resources in large-scale data environments. Taking product queries on e-commerce platforms as an example, if the platform has millions or even tens of millions of product data entries, when users perform multi-condition queries (such as specific brands, price ranges, sales volumes, etc.), a single-node query may take several minutes or even longer to return results, severely impacting user experience and business efficiency. Furthermore, traditional query methods struggle to adapt to dynamically changing data environments and cannot promptly handle newly added data or changes in data structure.

[0004] In data partitioning, existing block-based algorithms often lack flexibility and adaptability. Algorithms with fixed block sizes cannot be adjusted according to data characteristics and actual needs, potentially resulting in data blocks that are too large or too small. Overly large data blocks increase the system burden during data processing and transmission; underly small data blocks incur excessive block management overhead, reducing overall system performance. Furthermore, traditional block-based algorithms typically do not consider data continuity and correlation, hindering subsequent data querying and analysis.

[0005] In the field of distributed storage, common storage node allocation algorithms have shortcomings in handling dynamic changes in nodes. When storage nodes join or leave the system, traditional algorithms may require large-scale data migration, which consumes a significant amount of network bandwidth and system resources, and may even cause temporary system paralysis. At the same time, existing algorithms struggle to achieve load balancing among storage nodes, easily leading to situations where some nodes are overloaded while others are idle, reducing the overall utilization of the system.

[0006] In terms of index building, traditional index structures are inefficient when handling large-scale, high-dimensional data. Simple single-level indexes cannot meet the needs of complex queries, and the construction and maintenance of multi-level indexes lack effective strategies, resulting in untimely index updates and unstable query performance. For example, in a medical data management system, the data contains various patient information (such as medical records, diagnosis results, treatment records, etc.), and traditional indexes struggle to quickly locate data that meets specific query conditions (such as all patients with a certain disease and treatment costs within a certain range). Summary of the Invention

[0007] The purpose of this invention is to provide a data partitioning and query processing system under a distributed architecture to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a data partitioning and query processing system under a distributed architecture, the system comprising: Data segmentation module: used to receive the raw data stream and divide the data into multiple data blocks based on an adaptive segmentation algorithm. The adaptive segmentation algorithm dynamically adjusts the block size and boundary markers according to the data characteristics. Distributed storage module: It uses a consistent hashing algorithm to distribute data blocks to multiple storage nodes and dynamically adjusts the distribution of virtual nodes of the hash ring according to the node load; Index building module: Based on the key attributes of data blocks, a global index table is generated through a multi-level inverted index algorithm. The global index table records the data block identifier, storage node location, and attribute association relationship. Query optimization module: Based on the user's query request, the original query statement is converted into a distributed set of subqueries using a query rewriting algorithm. The query rewriting algorithm generates subquery logic based on semantic parsing tree and index matching rules. Result aggregation module: Uses a parallel aggregation algorithm to deduplicate and sort the subquery results returned by multiple storage nodes to generate the final query result set.

[0009] Preferably, the adaptive block partitioning algorithm includes: A sliding window scan is performed on the original data stream to identify abrupt changes in data characteristics as block boundaries; The initial block size is calculated based on the data density between mutation points, and the window step size is dynamically adjusted based on historical block efficiency. Add a unique identifier and boundary check code to each data block to generate block metadata.

[0010] Preferably, the consistent hashing algorithm includes: The storage nodes are mapped to multiple virtual nodes on the hash ring, and each virtual node corresponds to a storage partition of the physical node; Calculate the hash value based on the data block identifier, and find the first virtual node as the target storage node by searching clockwise along the hash ring; When a node joins or leaves, only the data block mapping relationship between adjacent virtual nodes is adjusted.

[0011] Preferably, the multi-level inverted index algorithm includes: Extract key attribute values ​​from data blocks and construct a mapping relationship between attribute values ​​and data block identifiers; Based on the frequency distribution of attribute values, high-frequency attributes and low-frequency attributes are stored in a hierarchical manner, generating a primary index and an auxiliary index; The hierarchical indexes are integrated into a global index table through an index merging strategy.

[0012] Preferably, the query rewriting algorithm includes: Parse the semantic structure of the user's query statement and generate a syntax parse tree; The candidate storage node set is determined by matching the attribute conditions in the syntax parse tree with the global index table; The original query is split into multiple subqueries, and node filter conditions and aggregation tags are added to each subquery.

[0013] Preferably, the parallel aggregation algorithm includes: Redundancy detection is performed on the subquery results, and duplicate data is filtered based on unique identifiers; the deduplicated data blocks are sorted using multi-way merge sort according to the sorting rules specified by the user; and the sorted data blocks are assembled into the final result set according to the pagination rules.

[0014] Preferably, the method for identifying data feature mutation points includes: Calculate the statistical difference between adjacent windows in the data stream, including changes in mean, variance, and entropy. When the statistical difference exceeds a preset threshold, mark the end position of the current window as the block boundary; The sensitivity of mutation point detection is optimized through iterative updates using a sliding window.

[0015] Preferably, the virtual node distribution adjustment method includes: Monitor the storage load and response latency of each physical node and calculate the node weight; dynamically increase or decrease the number of virtual nodes on the hash ring based on the node weight; balance the distribution of data blocks through virtual node migration strategies.

[0016] Preferably, the index merging strategy includes: Perform an intersection operation on the primary index and the secondary index to filter the data block identifiers that satisfy multiple condition queries; Reduce index storage space by using bitmap compression technology, regularly clean up invalid index entries, and update the version number of the global index table.

[0017] Preferably, the method for generating the syntax parse tree includes: Perform word segmentation on user queries and identify query keywords and logical operators; Construct a tree structure with keywords as leaf nodes and logical operators as intermediate nodes; validate the validity of the tree structure through syntax rules and optimize the execution order of operators.

[0018] Compared with the prior art, the beneficial effects of the present invention are: In the data segmentation stage, the adaptive block partitioning algorithm dynamically adjusts the block size and boundary markers based on data characteristics. By using a sliding window to scan and identify abrupt changes in data characteristics as block boundaries, it accurately divides data blocks with similar features, ensuring the logical coherence of the data. For example, when processing financial transaction data, it can be partitioned based on changes in transaction type, facilitating targeted analysis of different types of transaction data later. The initial block size is calculated based on the data density between abrupt changes, and the window step size is dynamically adjusted in conjunction with historical partitioning efficiency, making the block size more reasonable. This not only reduces the overhead of data processing and transmission but also improves the efficiency of data storage and retrieval. Adding a unique identifier and boundary check code to each data block ensures the integrity and accuracy of data during storage and transmission, reducing the risk of data errors.

[0019] The distributed storage module employs a consistent hashing algorithm, mapping storage nodes to multiple virtual nodes on a hash ring. Each virtual node corresponds to a storage partition of a physical node. This approach ensures a more even distribution of data blocks, avoiding uneven load distribution among storage nodes. The target storage node is determined by calculating the hash value based on the data block identifier, a simple and efficient operation. When a node joins or leaves, only the data block mapping relationship between adjacent virtual nodes is adjusted, significantly reducing data migration and ensuring system stability and reliability. Taking a large-scale distributed file storage system as an example, the addition of a new node or the departure of an old node does not severely impact the data storage and access of the entire system, maintaining its normal operation.

[0020] The index building module utilizes a multi-level inverted index algorithm to generate a global index table. It constructs mapping relationships by extracting key attribute values ​​from data blocks and stores high-frequency and low-frequency attributes hierarchically based on their frequency distribution, generating primary and secondary indexes, thus improving query efficiency. An index merging strategy quickly filters data block identifiers that meet multiple query conditions, bitmap compression technology reduces index storage space, and invalid index entries are regularly cleaned up and version numbers are updated to ensure the index's timeliness and accuracy. When processing e-commerce product data queries, it can quickly locate product data blocks that meet multiple attribute conditions (such as brand, price, sales volume, etc.), significantly shortening query response time.

[0021] The query optimization module transforms the original query statement into a distributed set of subqueries using a query rewriting algorithm. Based on semantic parsing trees and index matching rules, it generates subquery logic that accurately analyzes user query intent, determines the candidate storage node set, and reduces unnecessary query scope. By splitting the original query into multiple subqueries and adding node filtering conditions and aggregation tags to each subquery, it achieves parallel query processing, fully utilizes the computing resources of the distributed system, and improves query efficiency. For example, in an Enterprise Resource Planning (ERP) system, complex business queries can be quickly decomposed and distributed to multiple storage nodes for processing, significantly improving data query speed.

[0022] The result aggregation module uses a parallel aggregation algorithm to deduplicate and sort the subquery results returned by multiple storage nodes, generating the final query result set. Redundancy detection filters duplicate data based on unique identifiers, ensuring the accuracy of the results. Multi-way merge sort is performed according to user-specified sorting rules to meet diverse user query needs. The final result set is assembled according to pagination rules for easy viewing and improved user experience. In the online travel booking system, when users query hotel information, they can quickly obtain accurate results that are deduplicated, sorted, and paginated, improving the system's usability and practicality.

[0023] This invention comprehensively enhances data processing and query capabilities under a distributed architecture, from data segmentation, storage, index construction, query optimization to result aggregation. It provides efficient and stable technical support for big data applications, reduces system costs, and improves business operation efficiency, thus possessing significant practical application value. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the working principle of the data segmentation and query processing system described in this invention. Figure 2 This is a diagram illustrating the working principle of the consistent hashing algorithm. Figure 3 This is a diagram illustrating the working principle of a multi-level inverted index algorithm. Figure 4 This is a diagram illustrating the working principle of the query rewrite algorithm. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figures 1-4This invention provides a data partitioning and query processing system under a distributed architecture, the overall implementation scheme of which is as follows: The system mainly includes a data partitioning module, a distributed storage module, an index building module, a query optimization module, and a result aggregation module. In practical application scenarios, suppose there is a massive business data processing scenario, such as the transaction record processing of an e-commerce platform, where data is constantly generated and needs to be stored and queried efficiently.

[0027] The data segmentation module receives raw data streams, such as real-time transaction record data streams generated by e-commerce platforms. This module divides the data into multiple data blocks based on an adaptive segmentation algorithm. The adaptive segmentation algorithm scans the raw data stream using a sliding window to identify abrupt changes in data characteristics as block boundaries, such as locations where product categories suddenly change in transaction records. It calculates the initial block size based on the data density between these abrupt changes, assigning smaller blocks to areas with high data density to ensure efficient data processing. The window step size is dynamically adjusted based on historical segmentation efficiency. A unique identifier and boundary check code are added to each data block to generate block metadata, facilitating subsequent data management and verification.

[0028] The distributed storage module uses a consistent hashing algorithm to distribute data blocks to multiple storage nodes. First, the storage nodes are mapped to multiple virtual nodes on a hash ring, with each virtual node corresponding to a storage partition of a physical node. Then, the hash value is calculated based on the data block identifier, and the first virtual node is searched clockwise along the hash ring as the target storage node. When a node joins or leaves, only the data block mapping relationship between adjacent virtual nodes is adjusted, reducing the amount of data migration and ensuring the stability and reliability of the system.

[0029] The index building module generates a global index table based on the key attributes of data blocks using a multi-level inverted index algorithm. Taking e-commerce transaction records as an example, key attributes could be product ID, transaction time, etc. The multi-level inverted index algorithm extracts the key attribute values ​​of data blocks and constructs a mapping relationship between attribute values ​​and data block identifiers; it stores high-frequency and low-frequency attributes hierarchically according to the frequency distribution of attribute values, generating primary and secondary indexes; and it integrates the hierarchical indexes into a global index table through an index merging strategy, improving query efficiency.

[0030] The query optimization module transforms the original query statement into a distributed set of subqueries based on the user's query request using a query rewriting algorithm. For example, if a user queries the transaction records of a specific product within a certain period, the query rewriting algorithm first parses the semantic structure of the user's query statement to generate a syntax parse tree; it then matches the attribute conditions in the syntax parse tree with the global index table to determine the candidate storage node set; finally, it splits the original query into multiple subqueries and adds node filtering conditions and aggregation tags to each subquery, thus achieving distributed query optimization.

[0031] The result aggregation module uses a parallel aggregation algorithm to deduplicate and sort the subquery results returned by multiple storage nodes, generating the final query result set. When processing e-commerce transaction record query results, the parallel aggregation algorithm performs redundancy detection on the subquery results, filtering duplicate data based on unique identifiers; according to user-specified sorting rules, such as sorting by transaction time, it performs multi-way merge sorting on the deduplicated data blocks; and assembles the sorted data blocks into the final result set according to pagination rules for easy user viewing.

[0032] The technical solution of the present invention will be further described in detail below through specific embodiments. Example

[0033] In this embodiment, a financial transaction data processing system is used as an example to illustrate the adaptive block partitioning algorithm and the method for identifying data feature mutation points.

[0034] Financial transaction data is generated in real time, and the data segmentation module begins processing the raw data stream. The first step is identifying abrupt changes in data characteristics. The system uses a sliding window with a size of W and a step size of S. While scanning the data stream, the statistical difference between adjacent windows is calculated. Assume the data in the current window is... The data in the next window is Calculate the mean difference The formula is ,in Display window The mean of the data, Display window The mean of the data; calculation of variance variability. The formula is , Represents window data The variance; calculate the entropy value difference. Entropy calculation is based on information theory formulas. ,in It is data The Middle The probability of taking a certain value. It is the number of different values. Combining these three statistical differences, we obtain the total difference. , , , These are weighting coefficients, set according to data characteristics and business needs, and generally range in the range of... arrive Between, and .

[0035] When the total difference Exceeding the preset threshold At this point, the end position of the current window is marked as the block boundary. For example, in financial transaction data, when the transaction type suddenly changes, the mean, variance, and entropy values ​​will change significantly, causing the total variance to exceed the threshold, at which point a block boundary is determined.

[0036] After determining the block boundaries, the initial block size is calculated. The initial block size is calculated based on the data density between abrupt change points. The calculation method is as follows Assuming the amount of data between two mutation points is The range of data on a certain key attribute (such as transaction amount) is ,but If the data density is high, it indicates that the data is relatively concentrated, and the initial block size... The corresponding settings are relatively small, such as , It is a constant set according to system performance and data processing requirements.

[0037] The window step size is dynamically adjusted based on historical block partitioning efficiency. The time of each block partitioning operation is recorded. and number of blocks Calculate block efficiency If the previous block partitioning was efficient, it indicates that the current window step size is appropriate and can be increased accordingly; if the block partitioning was inefficient, it suggests that the step size may be too large or too small and needs to be decreased. The step size adjustment formula is as follows: , This is the adjusted step size. It is the current step size. It's an adjustment factor, set based on experience. It is the average of historical block efficiency.

[0038] Finally, a unique identifier and boundary checksum are added to each data block to generate block metadata. The unique identifier can be generated using a combination of timestamps, random numbers, etc., to ensure uniqueness. The boundary checksum is obtained by hashing the data block content and is used to verify the integrity of the data block during storage and transmission. These steps achieve efficient data segmentation, laying the foundation for subsequent data storage and query processing. Example

[0039] Taking a distributed file storage system as an example, this paper details the consistent hashing algorithm and the method for adjusting the distribution of virtual nodes.

[0040] In a distributed file storage system, there are multiple storage nodes, such as nodes. ,node ,node First, the storage nodes are mapped to multiple virtual nodes on the hash ring. Assume each physical node is mapped... A virtual node, with node For example, its virtual node is represented as , ... Each virtual node corresponds to a physical node. A storage partition.

[0041] When storing data blocks, based on the data block identifier Calculate hash value The hash function can employ common hash algorithms such as MD5 and SHA-1. Then, the first virtual node is searched clockwise along the hash ring as the target storage node. For example, the hash value of the data block identifier is calculated. The position on the hash ring is located in the virtual node. and Between them, the first virtual node searched clockwise is Then the data block is stored in the corresponding node. Storage partition.

[0042] When a node joins or leaves, only the data block mapping relationship between adjacent virtual nodes is adjusted. Assume a node... Once joined to the system, the system will assign it a virtual node. , ... Then insert it into the hash ring. At this point, adjustments are needed. The data block mapping relationship between adjacent virtual nodes will change the original mapping to The hash values ​​of data blocks in clockwise adjacent virtual nodes are recalculated to determine whether migration is necessary. .

[0043] To achieve dynamic adjustment of virtual node distribution, the system needs to monitor the storage load and response latency of each physical node. Storage load This can be measured by the ratio of used storage capacity to total storage capacity, i.e. , This refers to the storage capacity already used by the node. This refers to the total storage capacity of the node. Response latency. This can be obtained by recording the time it takes for a node to process a request, such as the time from receiving a request to returning a response.

[0044] Calculate node weights The formula is , and This is a weighting coefficient, set according to the system's emphasis on storage load and response latency, with a value ranging from... arrive Between, and The number of virtual nodes on the hash ring can be dynamically increased or decreased based on node weights. If a node has a high weight, indicating low load and fast response, its virtual node count can be appropriately increased. , This is the adjusted number of virtual nodes, and this is the current number of virtual nodes. It is an adjustment factor. It is the average weight of all nodes; otherwise, the number of virtual nodes is reduced.

[0045] A virtual node migration strategy is used to balance data block distribution. When the number of virtual nodes is adjusted, some data blocks need to be migrated from physical nodes with more virtual nodes to physical nodes with fewer virtual nodes. During migration, data blocks with lower access frequency are migrated first to reduce the impact on system performance. These methods ensure load balancing and stability of the distributed storage system. Example

[0046] Taking the product database management scenario of a large e-commerce platform as an example, this paper elaborates on the specific implementation of the multi-level inverted index algorithm and index merging strategy.

[0047] This e-commerce platform generates massive amounts of product data daily, including basic product information (such as product ID, name, category, brand, etc.), price information, inventory information, and sales data. This data is stored in a distributed database in the form of data blocks for easy management and querying.

[0048] The multi-level inverted index algorithm comes into play. First, it extracts the key attribute values ​​of the data blocks and constructs a mapping relationship between attribute values ​​and data block identifiers. For example, the product ID is a very important key attribute; for each product, its product ID is a unique identifier. Assuming the product ID "001001" is stored in data block "DB007", then a mapping relationship of "001001→DB007" is established. At the same time, product category is also a commonly used key attribute. For example, multiple products under the category "clothing" may be stored in different data blocks. For the category attribute value of "clothing", the identifiers of all data blocks containing products of that category are recorded.

[0049] Based on the frequency distribution of attribute values, high-frequency and low-frequency attributes are stored hierarchically, generating primary and secondary indexes. In e-commerce data, attributes such as product category and brand have relatively high query frequency and are considered high-frequency attributes. Their mapping relationships are stored in the primary index, which speeds up common queries. For example, when a user searches for "clothing of a certain brand," the primary index can quickly locate the set of data block identifiers containing clothing products of that brand. On the other hand, some special attributes, such as limited edition numbers or specific promotional activity identifiers, appear less frequently and are considered low-frequency attributes. Their related mapping relationships are stored in secondary indexes. This approach improves the efficiency of high-frequency attribute queries without negatively impacting the overall index performance due to the large amount of storage for low-frequency attributes.

[0050] Next is the index merging strategy. In actual queries, users often submit multi-condition queries, such as "search for clothing from a certain brand with prices between 100 and 500 yuan". In this case, the primary and secondary indexes are intersected. The primary index is then used to select the set of data block identifiers containing clothing items from that brand. Since price is a low-frequency attribute, the secondary index is used to filter out a set of data block identifiers with prices between 100 and 500 yuan. Through intersection operation This will give you a set of data block identifiers that satisfy multiple query conditions. The products stored in these data blocks are the products that meet the user's query conditions.

[0051] To reduce index storage space, bitmap compression is used. In bitmap compression, for each attribute value, a binary bit is used to indicate whether the data block contains that attribute value. For example, for the "clothing" category, if the data block... If the product is clothing, the corresponding bit is set to 1; otherwise, it is 0. Through compression algorithms such as Run-Length Encoding (RLE), consecutive identical bits are compressed into a single combination of a representation bit and length. Assuming that the bits corresponding to the "clothing" category in 10 consecutive data blocks are all 1, after RLE compression, they can be represented by "1 (10)," significantly reducing storage space.

[0052] Regularly cleaning up invalid index entries is also crucial. When a product data block is deleted, or certain product attributes change and no longer meet the original index criteria, their related mappings in the index become invalid index entries. The system periodically scans the index, deletes these invalid entries, and updates the version number of the global index table. This ensures that queries always use the latest and most accurate index information, guaranteeing the accuracy and efficiency of query results. Through these multi-level inverted index algorithms and index merging strategies, e-commerce platforms can quickly respond to various user query requests, improving user experience and business operational efficiency. Example

[0053] Taking an enterprise-level customer relationship management (CRM) system as an example, this paper details the query rewriting algorithm and the method for generating a syntax parse tree.

[0054] This CRM system stores a large amount of interaction data between businesses and customers, including basic customer information (such as company name, contact person, contact information, etc.), business transaction records (such as order information, communication records, cooperative projects, etc.), and customer service data (such as complaint records, after-sales support, etc.). When using the CRM system, users will submit various complex queries based on different business needs.

[0055] The query rewriting algorithm begins by parsing the semantic structure of the user's query. For example, if a user inputs "Query customer information for those who have done business with ABC Company after January 1, 2024, and whose purchases exceed 50,000 yuan," the system uses natural language processing technology and word segmentation tools to segment this query, identifying the query keywords "after January 1, 2024," "ABC Company," "have done business with," "purchases exceed 50,000 yuan," "customer information," and the logical operator "and."

[0056] Construct a tree structure with keywords as leaf nodes and logical operators as intermediate nodes. In this example, "AND" acts as the logical operator, becoming the intermediate node of the tree structure. Its left and right child nodes are the conditions "have business dealings with ABC Company after January 1, 2024" and "purchased products exceeding 50,000 yuan," respectively. Each sub-condition can be further subdivided. For example, "have business dealings with ABC Company after January 1, 2024" can be subdivided into the time condition "after January 1, 2024," the customer condition "ABC Company," and the business relationship condition "have business dealings." This tree structure clearly displays the logical relationships of the query statement.

[0057] The system validates the tree structure's legality using syntax rules and optimizes operator execution order. It checks the correctness of logical operators, the syntacticity of conditions for left and right child nodes, and the existence of contradictory or unreasonable condition combinations. When optimizing operator execution order, the system considers the distribution of attributes and data storage characteristics in the global index table. If the condition "purchased products exceeding 50,000 yuan" can quickly locate a small data block in the index, then the subquery corresponding to this condition is executed first, thus quickly narrowing the query scope and improving query efficiency.

[0058] The system determines a set of candidate storage nodes by matching attribute conditions in the parse tree with the global index table. Assume the global index table records the correspondence between attributes such as customer information, transaction time, and purchase amount for each data block and the corresponding storage nodes. By matching various conditions in the parse tree, the system can determine which storage nodes might contain data blocks that meet the conditions, thus obtaining the set of candidate storage nodes.

[0059] The original query is broken down into multiple subqueries, and node filtering conditions and aggregation tags are added to each subquery. For example, for each node in the obtained candidate storage node set, the system generates a corresponding subquery. For a candidate storage node "Node003", the subquery might be "retrieve customer information from Node003 that meets the criteria of having business dealings with ABC Company after January 1, 2024, and having purchased products exceeding 50,000 yuan". An aggregation tag is added for unified processing in the result aggregation module. This query rewriting algorithm transforms the complex original query into multiple distributed subqueries, fully utilizing the computing resources of the distributed system, improving the parallel processing capability of queries, and significantly shortening query response time, thus meeting the needs of enterprises for rapid querying when processing large amounts of customer data. Example

[0060] Taking an online travel booking platform as an example, this paper details the application of parallel aggregation algorithms in real-world scenarios.

[0061] This online travel booking platform integrates information on various travel products such as hotels, air tickets, and tourist attraction tickets from around the world, receiving a large number of user query requests every day. When a user makes a query, such as "search for hotels in the Paris area with a rating of 4 stars or higher, sorted by price from low to high, displaying 15 results per page," the system sends the query request to various storage nodes for subquery processing, and multiple storage nodes return their respective subquery results.

[0062] The parallel aggregation algorithm first performs redundancy checks on the subquery results, filtering duplicate data based on unique identifiers. In travel booking platforms, a hotel's unique identifier can be a globally unique code (such as a code assigned by an international hotel alliance or the platform itself). Each hotel record in the subquery results carries this unique identifier. By comparing these identifiers, the system can quickly identify and filter out duplicate hotel records. For example, due to data redundancy or data synchronization issues across different storage nodes, the same hotel might appear in multiple subquery results. By comparing unique identifiers, only one record is retained, avoiding duplicate displays to the user and ensuring the accuracy of the query results.

[0063] Perform a multi-way merge sort on the deduplicated data blocks according to the user-specified sorting rules. In this example, the user requests sorting by price from lowest to highest. Assume there are a total of Each storage node returns the subquery results, and each result set is... , ... The hotel records in each result set have been preliminarily sorted by price (this is because during the subquery phase, storage nodes can perform preliminary sorting based on the characteristics of local data, improving subsequent aggregation efficiency). A multi-way merge algorithm is used, comparing prices starting from the first record in each result set. For example, The first record hotel The price is EUR, The first record hotel The price is EUR, The first record hotel The price is If it's in Euros, then choose the hotel with the lowest price. Add the hotel to the final sorted result set. Result set The next record is added for comparison, and so on, until all records in the result set have been processed. This results in a list of hotels sorted from lowest to highest price.

[0064] The sorted data blocks are assembled into the final result set according to pagination rules. In this example, each page displays 15 records. The system starts with the first record in the sorted hotel list, and each page contains 15 records. The first page contains the 1st to 15th sorted hotel records, the second page contains the 16th to 30th records, and so on. This pagination assembly method allows users to easily view hotel information matching their criteria page by page when browsing the query results, improving the user experience. Simultaneously, this parallel aggregation algorithm fully utilizes the parallel processing capabilities of the distributed system, quickly processing the subquery results from multiple storage nodes, improving the efficiency of query result generation, and meeting the high concurrency and real-time business needs of online travel booking platforms.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data partitioning and query processing system under a distributed architecture, characterized in that, include: Data segmentation module: used to receive the raw data stream and divide the data into multiple data blocks based on an adaptive segmentation algorithm. The adaptive segmentation algorithm dynamically adjusts the block size and boundary markers according to the data characteristics. Distributed storage module: It uses a consistent hashing algorithm to distribute data blocks to multiple storage nodes and dynamically adjusts the distribution of virtual nodes of the hash ring according to the node load; Index building module: Based on the key attributes of data blocks, a global index table is generated through a multi-level inverted index algorithm. The global index table records the data block identifier, storage node location, and attribute association relationship. Query optimization module: Based on the user's query request, the original query statement is converted into a distributed set of subqueries using a query rewriting algorithm. The query rewriting algorithm generates subquery logic based on semantic parsing tree and index matching rules. Result aggregation module: Uses a parallel aggregation algorithm to deduplicate and sort the subquery results returned by multiple storage nodes to generate the final query result set.

2. The data partitioning and query processing system under a distributed architecture according to claim 1, characterized in that, The adaptive block partitioning algorithm includes: A sliding window scan is performed on the original data stream to identify abrupt changes in data characteristics as block boundaries; The initial block size is calculated based on the data density between mutation points, and the window step size is dynamically adjusted based on historical block efficiency. Add a unique identifier and boundary check code to each data block to generate block metadata.

3. The data partitioning and query processing system under a distributed architecture according to claim 1, characterized in that, The consistent hashing algorithm includes: The storage nodes are mapped to multiple virtual nodes on the hash ring, and each virtual node corresponds to a storage partition of the physical node; Calculate the hash value based on the data block identifier, and find the first virtual node as the target storage node by searching clockwise along the hash ring; When a node joins or leaves, only the data block mapping relationship between adjacent virtual nodes is adjusted.

4. The data partitioning and query processing system under a distributed architecture according to claim 1, characterized in that, The multi-level inverted index algorithm includes: Extract key attribute values ​​from data blocks and construct a mapping relationship between attribute values ​​and data block identifiers; Based on the frequency distribution of attribute values, high-frequency attributes and low-frequency attributes are stored in a hierarchical manner, generating a primary index and an auxiliary index; The hierarchical indexes are integrated into a global index table through an index merging strategy.

5. The data partitioning and query processing system under a distributed architecture according to claim 1, characterized in that, The query rewriting algorithm includes: Parse the semantic structure of the user's query statement and generate a syntax parse tree; The candidate storage node set is determined by matching the attribute conditions in the syntax parse tree with the global index table; The original query is split into multiple subqueries, and node filter conditions and aggregation tags are added to each subquery.

6. The data partitioning and query processing system under a distributed architecture according to claim 1, characterized in that, The parallel aggregation algorithm includes: Redundancy detection is performed on the subquery results, and duplicate data is filtered based on unique identifiers; the deduplicated data blocks are sorted using multi-way merge sort according to the sorting rules specified by the user; and the sorted data blocks are assembled into the final result set according to the pagination rules.

7. A data partitioning and query processing system under a distributed architecture according to claim 2, characterized in that, The method for identifying data feature mutation points includes: Calculate the statistical difference between adjacent windows in the data stream, including changes in mean, variance, and entropy. When the statistical difference exceeds a preset threshold, mark the end position of the current window as the block boundary; The sensitivity of mutation point detection is optimized through iterative updates using a sliding window.

8. A data partitioning and query processing system under a distributed architecture according to claim 3, characterized in that, The virtual node distribution adjustment method includes: Monitor the storage load and response latency of each physical node and calculate the node weight; dynamically increase or decrease the number of virtual nodes on the hash ring based on the node weight; balance the distribution of data blocks through virtual node migration strategies.

9. A data partitioning and query processing system under a distributed architecture according to claim 4, characterized in that, The index merging strategy includes: Perform an intersection operation on the primary index and the secondary index to filter the data block identifiers that satisfy multiple condition queries; Reduce index storage space by using bitmap compression technology, regularly clean up invalid index entries, and update the version number of the global index table.

10. A data partitioning and query processing system under a distributed architecture according to claim 5, characterized in that, The method for generating the syntax parse tree includes: Perform word segmentation on user queries and identify query keywords and logical operators; Construct a tree structure with keywords as leaf nodes and logical operators as intermediate nodes; validate the validity of the tree structure through syntax rules and optimize the execution order of operators.

Images