A query optimization method and system for distributed database

Abstract

The present application provides a query optimization method and system for a distributed database. The method includes: determining a high-frequency data unit set in a distributed database, and generating a sub-query task division scheme for each data unit in the high-frequency data unit set; determining the interaction frequency between each data unit according to the dependency relationship between sub-query tasks represented by the sub-query task division scheme, and determining a data distribution scheme based on the interaction frequency; determining an index set for each node to query the stored data units according to the storage relationship indicated by the data distribution scheme; for each index set, calculating the execution cost of different sub-query tasks according to the access mode supported by the data units supported by the index set, and generating a sub-query task parallel scheme based on the execution cost; generating a node scheduling scheme that satisfies the execution of the sub-query task parallel scheme. The present application can improve the query speed of a distributed database and the overall performance of the system.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a query optimization method and system for a distributed database. Background Art

[0002] With the rapid development of information technology, distributed databases, as a core technology for modern data management, play a vital role in processing massive amounts of data, supporting high-concurrency access, and achieving system scalability. Especially with the rapid adoption of cloud computing and big data applications, the query performance of distributed databases is directly related to system efficiency and user experience. However, existing query optimization methods exhibit significant limitations when dealing with complex query requirements and highly dynamic data access patterns.

[0003] Many traditional optimization solutions rely on static partitioning or simple load balancing strategies. These methods are difficult to adapt to the dynamic changes in data access patterns, resulting in frequent cross-node data transmission and increased query latency. In addition, existing technologies lack an effective mechanism for the coordination between data distribution optimization and query decomposition, which limits the improvement of the overall system performance. Especially in the field of distributed database query optimization, how to efficiently organize data distribution and optimize the query execution process is one of the core challenges. For example, the uneven distribution of frequently accessed data units will increase the cost of cross-node communication, and the decomposition and parallel execution of complex queries lack refined management, making it difficult to fully utilize the computing power of nodes. At the same time, the dynamic evaluation of data similarity between nodes and the inefficiency of index construction also restrict the adaptability of query optimization.

[0004] Therefore, the present application provides a query optimization method and system for a distributed database to solve one of the above technical problems. Summary of the Invention

[0005] The purpose of this application is to provide a distributed database query optimization method and system that can solve at least one of the above-mentioned technical problems. The specific solution is as follows:

[0006] According to a specific embodiment of the present application, in a first aspect, the present application provides a query optimization method for a distributed database, comprising:

[0007] A high-frequency data unit set is determined in a distributed database, and a sub-query task division scheme is generated for each data unit in the high-frequency data unit set; based on the dependency relationship between the sub-query tasks represented by the sub-query task division scheme, the interaction frequency between the data units is determined, and a data distribution scheme is determined based on the interaction frequency; wherein, the data distribution scheme is used to indicate the storage relationship of each data unit in each node; based on the storage relationship indicated by the data distribution scheme, an index set for each node to query the stored data units is determined; for each index set, the execution cost of different sub-query tasks is calculated respectively according to the access mode supported by the data unit supported by the index set for query, and a sub-query task parallel scheme is generated based on the execution cost; wherein, the sub-query task parallel scheme is used to indicate the required number of nodes, and one or more sub-query tasks required to be processed by each node; a node scheduling scheme that satisfies the execution of the sub-query task parallel scheme is generated.

[0008] According to a specific embodiment of the present application, in a second aspect, the present application provides a query optimization system for a distributed database, comprising:

[0009] A generation unit is used to determine a high-frequency data unit set in a distributed database and generate a sub-query task division scheme for each data unit in the high-frequency data unit set; a determination unit is used to determine the interaction frequency between the data units based on the dependency relationship between the sub-query tasks represented by the sub-query task division scheme, and determine the data distribution scheme based on the interaction frequency; wherein the data distribution scheme is used to indicate the storage relationship between each data unit and each node; and is used to determine the index set of each node for querying the stored data units based on the storage relationship indicated by the data distribution scheme; the generation unit is also used to: for each of the index sets, calculate the execution cost of different sub-query tasks respectively according to the access mode supported by the data unit supported by the index set for query, and generate a sub-query task parallel scheme based on the execution cost; wherein the sub-query task parallel scheme is used to indicate the required number of nodes, and one or more sub-query tasks required to be processed by each node; and is used to generate a node scheduling scheme that meets the requirements of executing the sub-query task parallel scheme.

[0010] According to a specific embodiment of the present application, in a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement any one of the methods described in the first aspect.

[0011] According to the specific implementation of the present application, in a fourth aspect, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the method described in any one of the first aspects when the computer program / instruction is executed by a processor.

[0012] Compared with the prior art, the above solution of the embodiment of the present application has at least the following beneficial effects:

[0013] This application provides a query optimization method for a distributed database. By determining a set of high-frequency data units in a distributed database and generating a sub-query task partitioning scheme, this method can accurately identify the most frequently accessed data units in the system, thereby optimizing the storage and access methods of these data in a targeted manner. This not only improves query efficiency, but also reduces unnecessary resource consumption. The frequency of interaction between data units is determined based on the dependency relationship between sub-query tasks, and a data distribution plan is formulated accordingly, so that related data is stored on the same node as much as possible, greatly reducing the need for cross-node data transmission and further improving query speed and overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart showing a query optimization method for a distributed database is shown;

[0015] Figure 2 A flow chart of a method for generating a sub-query task partitioning scheme is shown;

[0016] Figure 3 A flow chart of a method for determining the frequency of interaction between data units is shown;

[0017] Figure 4 A flow chart of a method for generating a sub-query task partitioning scheme is shown;

[0018] Figure 5 A flow chart of a method for generating a node scheduling solution is shown;

[0019] Figure 6 A unit block diagram of a query optimization system for a distributed database according to an embodiment of the present application is shown;

[0020] Figure 7 The present invention is a block diagram of an electronic device for query optimization of a distributed database according to an exemplary embodiment. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0022] The terms used in the examples of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in the examples of this application and the appended claims are also intended to include plural forms, and unless the context clearly indicates otherwise, "a plurality" generally includes at least two.

[0023] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0024] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0025] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0026] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0027] It should be noted in particular that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0028] The optional embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The embodiment provided in this application is an embodiment of a query optimization method for a distributed database.

[0030] The following combination Figure 1 The embodiments of the present application are described in detail.

[0031] Figure 1 A flow chart of a distributed database query optimization method is shown. Figure 1 As shown, it includes steps S101 to S105.

[0032] Step S101 : determining a high-frequency data unit set in a distributed database, and generating a sub-query task division scheme for each data unit in the high-frequency data unit set.

[0033] Step S102 : determining the interaction frequency between the data units according to the dependency relationship between the sub-query tasks represented by the sub-query task division scheme, and determining the data distribution scheme according to the interaction frequency.

[0034] The data distribution scheme is used to indicate the storage relationship of each data unit in each node.

[0035] Step S103: determining an index set for each node to query the stored data units according to the storage relationship indicated by the data distribution scheme.

[0036] Step S104 : for each index set, respectively calculate the execution costs of different sub-query tasks according to the access modes supported by the data units supported by the index set for query, and generate a sub-query task parallel solution according to the execution costs.

[0037] The subquery task parallel scheme is used to indicate the required number of nodes and one or more subquery tasks that each node needs to process.

[0038] Step S105: Generate a node scheduling solution that satisfies the parallel solution for executing sub-query tasks.

[0039] The query optimization method for a distributed database provided by this application can accurately identify the most frequently accessed data units in the distributed database and generate a sub-query task division scheme by determining the set of high-frequency data units in the distributed database, thereby optimizing the storage and access methods of these data in a targeted manner. This not only improves query efficiency, but also reduces unnecessary resource consumption. The frequency of interaction between data units is determined based on the dependency relationship between sub-query tasks, and a data distribution scheme is formulated accordingly, so that related data is stored on the same node as much as possible, greatly reducing the need for cross-node data transmission and further improving query speed and overall system performance.

[0040] In the embodiment of the present application, the high-frequency data unit set may be determined in the following manner.

[0041] In some embodiments, access frequency statistics of each data unit are extracted from the access log of the distributed database, a time decay factor is applied to reduce the access weight of historical data in the access frequency statistics, and based on abnormal filtering and normalization processing, the access heat value of each data unit is obtained, and a heat distribution map is generated for the access mode type to which the data unit belongs, and a high-frequency data unit set is determined based on the heat distribution map.

[0042] Among them, access logs are obtained from the distributed database. For example, log parsing tools can be used to extract access records of each data unit, and a structured access data set is generated based on preset data unit identification rules to obtain a log data set containing data unit identification and access timestamps.

[0043] In some embodiments, the time decay factor formula can be applied by Indicates that H(t) represents the access weight after attenuation, C represents the original number of visits, λ represents the time attenuation coefficient, t represents the time difference between the current time and the access time, and e represents the base of the natural logarithm.

[0044] In some embodiments, the access weight of historical data in the access frequency statistics is reduced to obtain the attenuated access frequency. If the attenuated access frequency exceeds the preset abnormal threshold, the abnormal access is filtered out using the anomaly detection algorithm based on K-means clustering. For example, the minimum-maximum normalization formula is used for the filtered access frequency. Determine the normalized popularity value, where N represents the normalized popularity value, ranging from 0 to 1, F represents the current visit interaction frequency value, F_min represents the minimum value among all frequencies, and F_max represents the maximum value among all frequencies. This formula can map visit frequency data of different ranges into a unified interval, facilitating subsequent analysis and comparison to obtain the normalized visit popularity value.

[0045] On this basis, a hierarchical clustering algorithm can be used to classify the access patterns of data units based on the normalized access heat values. Based on the classification results, a heat distribution map is generated using a visualization tool. For data units in the heat distribution map whose heat values ​​exceed a preset threshold, a set of high-frequency data units is identified. In the heat distribution map, the distribution is based on the access pattern types supported by the data units, and the heat is based on the sum of the access heat values ​​of each data unit under a single access pattern type.

[0046] In some specific embodiments, obtaining access logs from distributed databases is fundamental to analyzing access patterns of data units. For example, an e-commerce platform stores user browsing history for products. Logs contain user IDs, product IDs, and timestamps. Log parsing tools can use regular expressions to extract these fields and generate structured datasets.

[0047] For example, each row in the parsed dataset might look like: Product ID P001, access time 2025-04-23 10:00:00. Compared to manual parsing, automated tools significantly improve efficiency and ensure data consistency. Structured datasets are generated based on pre-set data unit identification rules. These rules typically center around a unique identifier, such as the product ID, which serves as the data unit identifier. For example, product P001 represents a data unit, where P001 represents the data unit identifier.

[0048] In one feasible implementation, a rule can be defined as starting with the letter P followed by three digits to filter out invalid records. For example, P001 and P002 are valid product IDs. After parsing, a log data set containing product IDs and timestamps is formed. This regularized design facilitates subsequent statistics and analysis, reducing data redundancy. For the log data set, a counting algorithm calculates the access frequency of each data unit. Specifically, P001 was accessed 100 times and P002 was accessed 50 times within 24 hours. A time decay factor formula is applied to reduce the access weight of historical data in the access frequency statistics.

[0049] Preferably, the decay coefficient λ is set to 0.1, and the time difference t is in hours. The weight of older access records is reduced to reflect recent access trends. This method makes the access frequency closer to real-time user interests and improves the timeliness of analysis. If the decaying access frequency exceeds the abnormal threshold, the anomaly detection algorithm based on K-means clustering can identify abnormal access. For example, the decay frequency of P001 is 80, which exceeds the threshold of 50, and the surge in access may be caused by promotional activities. K-means clustering divides access patterns into normal and abnormal categories, and retains real user behavior data after filtering out anomalies. This filtering mechanism effectively eliminates the impact of crawlers or malicious access and ensures data reliability. For the filtered access frequency, the minimum-maximum normalization formula maps the frequency to the range of 0 to 1.

[0050] In some embodiments, the frequency of P001 is 80, P002 is 40, the minimum frequency F_min is 40, the maximum frequency F_max is 80, and the normalized heat value of P001 is 1, and that of P002 is 0. This normalization facilitates the comparison of the relative heat of different data units, eliminates dimensional differences, and improves visualization effects. Based on the normalized access heat values, the hierarchical clustering algorithm classifies the access patterns of the data units. It is understandable that the algorithm divides products into high-heat, low-heat, and other groups based on the similarity of heat values. For example, P001 and P003 are divided into high-heat groups, reflecting similar user preferences. The classification results generate a heat distribution map through a visualization tool, and the hot spots intuitively display the frequently visited products. This visualization helps the operation team quickly identify popular products and optimize the recommendation strategy.

[0051] For example, a high-frequency data unit cluster is identified for data units in the heat distribution graph whose heat values ​​exceed a preset threshold. For example, P001 and P003, whose heat values ​​are greater than 0.8, constitute a high-frequency cluster. These products can be prioritized for marketing promotions, significantly improving conversion rates. It should be noted that dynamic updates to the high-frequency cluster facilitate real-time adjustments to inventory and advertising strategies, enhancing platform operational efficiency.

[0052] In the embodiment of the present application, a sub-query task division scheme may be generated in the following manner.

[0053] Figure 2 A flow chart of a method for generating a sub-query task partitioning scheme is shown, as Figure 2 As shown, it includes steps S201 to S204.

[0054] Step S201 , obtaining the classification label of each data unit, extracting the predicate relationship corresponding to the access mode supported by each data unit, and generating a predicate relationship set including the predicate relationship and the data unit identifier through a preset predicate template matching algorithm.

[0055] Step S202 : extracting predicate dependency relationships from the predicate relationship set, and constructing a dependency relationship network for predicate dependency relationships with connection strengths higher than a preset threshold.

[0056] Step S203 : dividing each data unit into sub-query task granularities according to the dependency network, and generating an initial query set through a preset granularity control rule.

[0057] Step S204 : sorting the access frequencies of the sub-query tasks for the initial query set, and generating a sub-query task partitioning scheme by using a partitioning algorithm based on access frequencies.

[0058] In an embodiment of the present application, the classification labels of high-frequency data units are obtained from the heat distribution graph, the predicate relations corresponding to the access patterns of the data units are extracted using a logic parsing tool, and a structured relational data set containing predicate relations and data unit identifiers is generated through a preset predicate template matching algorithm to obtain a predicate relation set. Among them, for the predicate relation set, a graph parsing algorithm is used to extract the dependency relations between the predicate relations. If the connection strength of the dependency relations is higher than a preset threshold, a dependency network is constructed through a weighted directed graph generation tool to obtain a dependency network. According to the dependency network, the K-means clustering algorithm is used to divide the data units into sub-query task granularity, and the initial query set is generated through the preset granularity control rules. In addition, for the initial query set, an access frequency statistical data tool is used to sort the access frequency of each sub-query task, and a sub-query task division scheme is generated through a division algorithm based on access frequency to determine the sub-query task division scheme.

[0059] In the above-mentioned embodiment, when constructing a subquery task partitioning scheme, the classification labels of each data unit are obtained, the predicate relationships corresponding to the access patterns supported by the unit are extracted, and a predicate relationship set is generated using a predicate template matching algorithm. This method effectively captures the inherent connections between user behavior patterns. Furthermore, after establishing the dependency network, the subquery tasks are partitioned at a granular level based on dependency strength, ensuring that each subquery task covers key aspects of user behavior. At the same time, query partitioning is guided by access frequency ranking, prioritizing high-frequency tasks. This significantly improves system response speed and user experience.

[0060] In some specific embodiments, classification labels for high-frequency data units are obtained from a heat distribution map, and predicate relationships corresponding to access patterns can be extracted using a logical parsing tool. For example, on an e-commerce platform, a heat distribution map shows products P001 and P003 as high-frequency data units, with a classification label of "high popularity." The logical parsing tool analyzes user access logs and extracts predicate relationships such as "User U001 browsed P001."

[0061] In some specific embodiments, the tool generates triples of the form "subject-predicate-object" based on user behavior in logs. The predicate relationship of P001 may include "browse" and "favorite." This approach facilitates a structured description of user-product interactions.

[0062] As a feasible implementation method, a preset predicate template matching algorithm generates a structured relationship data set as a predicate relationship set.

[0063] For example, a template is defined as "user ID - behavior - product ID" and matches log records such as "U001 - browse - P001." The algorithm scans the logs, extracting all records matching the template and forming a dataset containing predicate relationships and product IDs. Preferably, the dataset is grouped by product ID to facilitate subsequent analysis of the distribution of predicate relationships. Within this set of predicate relationships, a graph parsing algorithm extracts dependency relationships.

[0064] For example, if P001's "browse" and "favorite" actions are dependent, the dependency strength is high if users frequently add items to favorites after browsing. Furthermore, the algorithm counts the number of occurrences of the "browse-favorite" sequence and calculates its strength. For example, P001's strength is 0.8, which is higher than the threshold of 0.6.

[0065] In the embodiment of the present application, a dependency relationship with high strength reflects the user behavior pattern, which is helpful for accurate recommendation. A dependency network can be constructed by a weighted directed graph generation tool. It can be understood that the nodes are predicates such as "browse" and "favorite", the edges are dependency relationships, and the weights are strength values. For example, the network of P001 shows that "browse" points to "favorite", with a weight of 0.8. This network intuitively displays the association between behaviors and facilitates the analysis of complex patterns. Based on the dependency network, the K-means clustering algorithm divides the sub-query task granularity. For example, the predicate relationship patterns of P001 and P003 are similar, and they are clustered into the same sub-query task group.

[0066] Preferably, the granularity control rule sets each sub-query task to include at least two predicate relationships to ensure that the query covers diverse behaviors. For example, the initial query set is in the form of "browse and collect records of P001".

[0067] As a feasible implementation, a tool for access frequency statistics ranks subquery tasks. For example, the query frequency for "browse and save P001" is 100, while the query frequency for "browse only P002" is 50. A frequency-based partitioning algorithm generates a subquery task partitioning scheme, prioritizing high-frequency queries, such as those related to P001. This scheme focuses on the access patterns of popular products, helping to optimize recommendations and inventory management.

[0068] Figure 3 A flow chart of a method for determining the interaction frequency between data units is shown. Figure 3 As shown, it includes steps S301 to S302.

[0069] Step S301 : obtaining data dependency relationships from the sub-query task partitioning scheme, and using the data dependency relationships as data dependency edges between the sub-query tasks to obtain a dependency graph between the sub-query tasks.

[0070] Step S302 : Calculate the interaction frequency between data units using a graph partitioning algorithm based on the dependency graph.

[0071] In this application, the data dependencies are extracted from the subquery task partitioning scheme and converted into a dependency graph. A graph partitioning algorithm is then used to calculate the interaction frequency between data units. This method provides a scientific basis for subsequent data distribution optimization. It not only helps identify which data units are closely related, but also helps understand the data flow of the entire system, laying the foundation for reducing unnecessary data transmission and improving query efficiency.

[0072] Figure 4 A flow chart of a method for generating a sub-query task partitioning scheme is shown, as Figure 2 As shown, it includes steps S401 to S402.

[0073] Step S401: Taking each interaction frequency as a matrix element, an interaction frequency matrix is ​​generated.

[0074] Step S402 : for the interaction frequency matrix, allocating data units corresponding to interaction frequencies higher than a preset threshold to the same node to obtain a data distribution solution.

[0075] In this embodiment, an interaction frequency matrix is ​​generated using each interaction frequency as a matrix element. Data units corresponding to interaction frequencies above a preset threshold are assigned to the same node. This approach effectively optimizes data locality and reduces cross-node communication costs. For data units with frequent interactions, this layout greatly improves query efficiency, simplifies system design, and enhances scalability and flexibility.

[0076] In some specific embodiments, extracting data dependencies from the subquery task partitioning scheme is key to optimizing data processing in e-commerce platform recommendation systems. Data dependencies reflect the data interaction patterns between subquery tasks, such as the associations between user behavior queries. Association analysis algorithms scan the subquery task results, extract frequent itemsets, and generate data dependency edges.

[0077] For example, subquery task Q1 is "records of users browsing item P001," and Q2 is "records of users adding item P001 to their favorites." Algorithm analysis logs reveal that the output data of Q1 and Q2 are often used simultaneously by subsequent recommendation modules, generating a dependency edge Q1→Q2. The weight is the number of co-occurrences, such as 80, reflecting a strong association. The dependency graph between subquery tasks uses subquery tasks as nodes and directed dependency edges to intuitively illustrate data flow. Based on the dependency graph between subquery tasks, the graph segmentation algorithm calculates the interaction frequency between data units. A data unit is an entity such as a product or user, and the interaction frequency refers to the number of co-occurrences of entities within a subquery task.

[0078] In some specific embodiments, the algorithm traverses the dependency graph and counts the number of times P001 and P003 co-occur in Q1 and Q2. For example, if P001 and P003 co-occur 50 times, an interaction frequency matrix is ​​generated. The matrix elements are interaction frequency values, such as 50 for P001-P003 and 20 for P001-P002. If the interaction frequency value is above a threshold of 30, the association strength is considered high.

[0079] Preferably, P001 and P003 are assigned to the same computing node to generate an initial data distribution plan, ensuring that high-frequency interactive data is processed on the same node and reducing cross-node communication.

[0080] As a feasible implementation, load balancing adjusts and optimizes the initial data distribution plan. A greedy algorithm evaluates the load of each computing node. For example, if node A processes P001 and P003 at a load of 100 queries per second, and node B processes P002 at a load of 50 queries per second, the algorithm migrates some low-frequency data, such as P002, to nodes with lower loads, generating an updated data distribution plan. It should be noted that this adjustment takes into account node storage capacity. For example, node A has a capacity of 500GB, which is sufficient to accommodate the data of P001 and P003. The updated data distribution plan ensures load balancing and consistent query efficiency across nodes. It is understood that the threshold determination of the interaction frequency matrix facilitates the identification of key data associations, and assigning them to the same node reduces query latency. The dynamic adjustment of the greedy algorithm adapts to changes in data volume and ensures system stability. For example, high-frequency queries for P001 are assigned to high-performance nodes, prioritizing user needs. This approach optimizes resource allocation by refining data dependency and interaction analysis, improving the responsiveness and accuracy of the recommendation system.

[0081] In an embodiment of the present application, the index set used by each node to query the stored data units can be determined in the following manner.

[0082] In some embodiments, a pre-established clustering algorithm can be used to classify data requests based on access patterns and classification rules, generating a hierarchical index framework and determining a hierarchical index structure for the access patterns. Based on data granularity and storage allocation, index entries are extracted from the hierarchical index framework and a hash table is used to construct a node-local set of index entries, resulting in an index set.

[0083] For example, processing access patterns and classification rules can be achieved by analyzing the characteristics of data requests. Access patterns refer to the query habits of users or systems for data, such as frequent access to certain fields or queries by time range. Classification rules are pre-defined based on dimensions such as request frequency, data type, or access time. Clustering algorithms, such as K-means clustering, can group requests by characteristics. For example, in a distributed database, suppose there are 10 million request logs containing user IDs, query fields, and timestamps. After analysis by the clustering algorithm, the requests are divided into high-frequency field query groups and low-frequency range query groups, generating a hierarchical index framework. This framework organizes the index in a tree structure, with high-frequency access patterns at the top level and low-frequency patterns at the bottom level, ensuring fast data location.

[0084] In some specific embodiments, the construction of the hierarchical index structure depends on the clustering results. For example, fields of high-frequency query groups such as "order amount" are assigned to the top-level nodes of the index tree, and the nodes store pointers to the data, while low-frequency queries such as "historical notes" are placed at the bottom. This structure facilitates the rapid retrieval of high-frequency data. Assuming that 80% of the 100,000 queries per day are concentrated in the order amount field, the top-level index can reduce the retrieval time by 90%. It should be noted that the hierarchical index framework needs to be updated regularly to adapt to changes in access patterns, such as re-clustering once a month.

[0085] As a feasible implementation, data granularity should be considered when extracting index entries from the hierarchical index framework. Data granularity refers to the size of the data shards, such as by day or by user ID. Assuming the data is sharded by day, each shard contains 100,000 records, and the index entry records the storage location of each shard. Hash tables are used to construct node-local index entry sets because they are highly efficient. For example, if a node manages 10 data shards, and the hash table uses the shard ID as the key and the index entry as the value, a query for the shard ID "2023-04-01" can directly locate it.

[0086] Preferably, the hash table needs to be set with an appropriate bucket size, such as 1000 buckets, to reduce the collision rate.

[0087] Among them, it is understandable that the generation of index sets needs to balance storage and query efficiency. For example, a node stores 1,000 index entries, each entry contains a shard ID and a pointer, and occupies about 10KB of memory. The average time taken for hash table queries is 1 microsecond, which is suitable for high-concurrency scenarios. In another embodiment, redundant index entries can be added for hot data, such as storing an additional index for the frequently accessed order amount field, pointing to shard copies of different nodes to improve fault tolerance. This method supports the goal of fast access through multi-faceted optimization, such as hot data priority, redundant indexes, and efficient hash table queries.

[0088] As a feasible implementation, an adaptive adjustment mechanism can be introduced to address dynamic access patterns. Suppose the query frequency for a field suddenly increases from a low frequency to a high frequency. Once detected, the system will promote its index entry to the top level. This mechanism ensures that the index framework always matches actual needs through the collaborative work of clustering algorithms and hash tables. It should be noted that adaptive adjustment must be performed at a controlled frequency, such as daily, to avoid frequent index rebuilds that waste resources. This approach progresses from access pattern analysis to index construction, layer by layer, to form an efficient data access system.

[0089] In some other embodiments, the data distribution scheme can be further adjusted by considering load balancing. For example, the data distribution scheme can be load-balanced and updated by optimizing computing resource allocation using a greedy algorithm. In the above embodiments, load balancing ensures a relatively balanced workload across all nodes, preventing overload on individual nodes. This not only fully utilizes system resources but also ensures efficient and stable system operation in high-concurrency scenarios, thereby improving system reliability and stability.

[0090] In an embodiment of the present application, a node scheduling scheme may be generated in the following manner.

[0091] Figure 5 A flow chart of a method for generating a node scheduling solution is shown in FIG. Figure 5 As shown, it includes steps S501 to S502.

[0092] Step S501 : According to the sub-query task parallel solution, a topological sorting algorithm is used to analyze the sub-query task dependencies, and a scheduling sequence is generated according to the sub-query task dependencies as the execution priority sequence of the sub-query tasks.

[0093] Step S502 : using the execution priority sequence as the allocation priority of the sub-query task, allocating the sub-query task to the corresponding node, and obtaining a node scheduling solution.

[0094] In this embodiment, a topological sorting algorithm is used to analyze subquery task dependencies and generate a scheduling order, enabling subquery tasks to be executed in the correct dependency order. This prevents data inconsistencies or query failures caused by incorrect ordering. This approach ensures that each subquery task is executed at the appropriate time, maximizing the use of system resources and improving query efficiency and accuracy.

[0095] In the embodiment of the present application, the node scheduling scheme may also be updated by one or a combination of the following methods.

[0096] Method 1: If the node scheduling plan satisfies the requirement that the target node's load rate exceed the preset load rate threshold, the resource allocation ratio for the target node is calculated, and the parallel granularity is adjusted based on the node heterogeneity between nodes to update the node scheduling plan. The target node represents one or more nodes that meet the threshold conditions.

[0097] Method 2: For the node scheduling plan, use the consistent hashing algorithm to analyze data distribution and data locality, adjust the allocation relationship between subquery tasks and nodes, and update the node scheduling plan.

[0098] In an embodiment of the present application, when the node scheduling scheme may cause the load rate of some nodes to be too high, by calculating the resource allocation rate of the target node and adjusting the parallel granularity in combination with the node heterogeneity, this method can dynamically adjust the task allocation strategy without affecting the overall performance of the system to achieve more reasonable resource allocation. In addition, the consistent hashing algorithm is used to analyze data distribution and data locality, further optimizing the allocation relationship between subquery tasks and nodes, ensuring the efficiency of data access and the system's scalability. This multi-level optimization measure works together to enable the system to perform well in the face of complex queries and highly dynamic data access patterns.

[0099] For example, in a distributed database, when analyzing dependencies among parallel subquery tasks, a topological sorting algorithm can be used to generate an ordered preliminary scheduling order. Topological sorting identifies dependencies between subquery tasks by constructing a directed acyclic graph. For example, an order query involves an order table and a user table, where the order status query depends on the user authentication query. The system models these two subquery tasks as graph nodes, with dependencies as edges. Topological sorting generates an order: user authentication is performed first, followed by the order status query. This approach ensures that subquery tasks are executed in the order of dependencies, preventing data inconsistencies.

[0100] As a feasible implementation method, based on the execution priority sequence generated by topological sorting, the system checks whether the node load rate is higher than a preset threshold, such as 80%. If the load rate of a node reaches 90%, the resource allocation rate is calculated through a hash table. The hash table uses the node ID as the key and the value is the CPU occupancy rate and memory usage. For example, the CPU occupancy rate of node A is 85% and the memory occupancy is 70%. Taking into account the heterogeneity of nodes, such as node A has an 8-core CPU and node B has a 16-core CPU, the system adjusts the parallel granularity and assigns the computationally intensive subquery task to node B, reducing the pressure on node A.

[0101] In some specific embodiments, the optimized node scheduling scheme needs to consider data distribution and data locality. A consistent hashing algorithm can be used to assign subquery tasks to corresponding nodes. For example, if order data shards are stored on nodes A and B, the system uses consistent hashing to calculate the hash value of the data key and map it to node A. Subqueries involving order status are preferentially assigned to node A, reducing cross-node data transmission. This approach leverages data locality and reduces network overhead.

[0102] The consistent hashing algorithm also supports dynamic adjustment. If a new node C is added, the system recalculates the hash ring, maps some data keys to node C, and redistributes subquery tasks accordingly. For example, after adding a new node, some subquery tasks for order status queries are assigned to node C, balancing the load. This approach adapts to node changes and ensures system scalability.

[0103] As a feasible implementation, the generated locally optimized node scheduling plan further refines execution details. For example, when processing an order status query, Node A prioritizes using local indexes to reduce disk I / O. Based on the execution plan, the system dynamically adjusts memory allocation for subquery tasks, such as allocating an additional 2MB of cache to high-priority subqueries. This refined management improves query efficiency.

[0104] It's clear that the above approach, from dependency analysis to task allocation, forms a complete execution optimization chain. Topological sorting ensures dependency correctness, while hash tables and consistent hashing algorithms collaboratively optimize resource allocation and data locality. Local optimization plans further enhance execution efficiency. This multi-layered collaborative approach supports high-concurrency query scenarios and adapts to dynamic changes in distributed environments.

[0105] In an embodiment of the present application, after generating a node scheduling scheme, index optimization can be further implemented according to the node scheduling scheme.

[0106] For example, in some embodiments, the data interaction requirements between nodes can be extracted from the node scheduling scheme. A kernel function algorithm can be used to calculate the similarity matrix of data units. If the similarity value exceeds a preset threshold, the data is marked as high-similarity data, resulting in similarity grouping based on data granularity. Furthermore, based on the similarity grouping, dynamic index adjustment can be used to update the local index storage of the node. If the index access efficiency exceeds a preset threshold, an optimized index set adapted for high concurrency is generated.

[0107] In the above embodiment, for example, inter-node data interaction requirements can be extracted from the node scheduling plan, and a graph database can be used to store the interaction relationships between nodes. The inter-node data interaction requirement set can be obtained by querying the edge weights between nodes in the graph database. Based on the inter-node data interaction requirement set, a kernel function algorithm is used to calculate the feature vector of each data unit to generate a similarity matrix. If the element value in the similarity matrix exceeds a preset threshold, the unit is marked as a high-similarity data unit, thereby obtaining a high-similarity data set. Based on the high-similarity data set, a clustering algorithm is used to group the high-similarity data units based on data granularity, thereby obtaining a similarity grouping set. Based on the data characteristics, a clustering algorithm is used to group the data based on similarity, obtaining the grouped data set, and determining the feature vectors of the data set. For the grouped data set, a B+ tree structure is used to generate a local index at the node, obtaining a local index set, and determining the storage format of the local index set. Based on the access pattern, the access frequency and distribution are obtained from the query log. A dynamic adjustment algorithm is used to update the local index set. The access efficiency of the local index set is determined to be above a preset threshold, thereby obtaining an adjusted index set. If the access efficiency of the adjusted index set is higher than a preset threshold, a hash table is used to generate an optimized index set, an index structure adapted to high concurrency is obtained, and the concurrency adaptability of the optimized index set is determined.

[0108] In some specific embodiments, in a distributed database, the analysis of data interaction requirements between nodes in a node scheduling solution is the key to optimizing query efficiency.

[0109] For example, the data interaction requirements between nodes can be modeled using a graph database. A graph database uses nodes to represent compute nodes, edges to represent data interactions, and edge weights to record the frequency of interactions or the amount of data. For example, an order query involves nodes A and B. Node A processes user data, and node B processes order data. The system calculates the interaction frequency and finds that node A transmits user identity data to node B 100 times per second. The edge weight is set to 100. The graph database is queried for edge weights to generate a data interaction requirement set, including 100 interactions per second between node A and node B. This approach clearly maps interaction relationships and facilitates subsequent optimization.

[0110] In some specific embodiments, based on a set of data interaction requirements, a kernel function algorithm is used to calculate the eigenvector of a data unit. The kernel function generates a eigenvector to represent its behavioral characteristics by analyzing the interaction pattern of the data unit. For example, the eigenvector of order data includes the interaction frequency value of the order ID, timestamp, and user ID. The system calculates a similarity matrix for the eigenvector, and the matrix element values ​​represent the similarity between the data units. If the preset threshold is 0.8, the data units with element values ​​higher than 0.8 in the matrix are marked as high similarity. For example, the data units with order IDs 1001 and 1002 have a similarity of 0.9 because their timestamps are close and the user IDs are the same, and are marked as high-similarity data units, constituting a high-similarity data set. This method effectively identifies the correlation between data.

[0111] Preferably, highly similar data sets are grouped by data granularity using a clustering algorithm. Clustering algorithms group similar data units based on their feature vectors. For example, the K-means clustering algorithm uses the Euclidean distance of feature vectors to classify order data into three groups: recent orders, historical orders, and high-frequency user orders. Each group has a different data granularity; for example, the recent orders group contains 1,000 records, which is a finer data granularity. After grouping, a similarity grouping set is generated, containing information about the three groups of order data. This grouping approach facilitates targeted query optimization.

[0112] As a feasible implementation, edge weight queries in graph databases support dynamic updates. For example, after adding node C, the system recalculates the interaction frequency between node A and node C and updates the edge weight to 50. The kernel function algorithm also adjusts the feature vector calculation logic to incorporate the data interaction pattern of node C. The clustering algorithm regroups the updated high-similarity data set to ensure that the grouping results adapt to system changes. This dynamic adjustment mechanism improves the scalability of the system. It is understandable that each link in the above method supports each other. The graph database clearly stores interaction relationships, the kernel function algorithm accurately extracts data features, and the clustering algorithm optimizes data grouping.

[0113] For example, in an order query scenario, the system uses a graph database to identify frequently interacting nodes. A kernel function algorithm generates an accurate similarity matrix. A clustering algorithm groups the order data, prioritizing queries based on high-frequency user orders and reducing unnecessary data scanning. This multi-layered collaborative optimization significantly improves query efficiency while adapting to the dynamic nature of distributed environments.

[0114] For example, in a distributed database system, clustering algorithms based on data features can be used to group users based on similarity. Suppose a distributed database stores user behavior data from an e-commerce platform. Data features include purchase frequency, browsing time, and product category preferences. Clustering algorithms such as K-means can group users based on these features, for example, grouping users who frequently purchase electronics. After grouping, a feature vector is generated for each group, reflecting the commonalities among users within the group. For example, the feature vector for a group might show that 90% of users prefer mobile phones and purchase an average of three times per month. This grouping facilitates subsequent index optimization and query acceleration.

[0115] As a feasible implementation method, the node locally generates a local index of a B+ tree structure for the grouped data set. B+ trees are suitable for distributed environments due to their efficient range query and sequential access characteristics. For example, a node stores the data of the above-mentioned high-frequency purchasing user group, and a B+ tree index can be constructed based on user ID or purchase time. The storage format of the local index set needs to take into account the node storage capacity and query requirements. It is preferred to use a compressed format to store non-leaf nodes to reduce I / O overhead. Assuming that the node stores 1 million records, the B+ tree index can reduce the query time from seconds for a full table scan to milliseconds.

[0116] In some specific embodiments, analyzing access patterns based on query logs is the key to optimizing indexes. Query logs record the frequency and distribution of user queries. For example, it was found that 60% of queries were concentrated on order data in the past week, and most of them were filtered by time range. The dynamic adjustment algorithm can update the local index set accordingly, such as allocating more index space to data in time periods with high frequency access. During the adjustment process, it is necessary to determine whether the access efficiency is higher than the preset threshold, such as requiring the average query delay to be less than 50 milliseconds. If the query delay of a node is reduced from 80 milliseconds to 40 milliseconds after the index of the node is adjusted, the threshold requirement is met.

[0117] As a feasible implementation, if the adjusted index set access efficiency meets the requirements, a hash table can be further used to generate an optimized index set to accommodate high-concurrency scenarios. Hash tables are suitable for precise match queries, such as quickly locating order records by user ID. Assuming the system needs to support 100,000 concurrent queries per second, a hash table can control the latency of a single query to microseconds. The concurrency adaptability of the optimized index set is reflected in its low conflict rate and high throughput. For example, by dynamically expanding the hash bucket capacity, 99% of queries are guaranteed to be conflict-free.

[0118] It's worth noting that the combination of hash tables and B+ trees can handle both range and point queries, significantly improving overall system performance. It's also clear that the aforementioned solution, through its progressive logic from clustering and grouping to index optimization, ensures efficient querying in distributed databases in high-concurrency scenarios. Each technical theme is implemented around a single scenario involving e-commerce user behavior data, ensuring the solution's focus and practicality.

[0119] The present application also provides a system embodiment that is consistent with the above embodiment, which is used to implement the method steps of the above embodiment. The explanation based on the same name meaning is the same as the above embodiment, and has the same technical effect as the above embodiment, which will not be repeated here.

[0120] like Figure 6 As shown, the present application provides a distributed database query optimization system 600, including:

[0121] The generating unit 601 is configured to determine a high-frequency data unit set in a distributed database and generate a sub-query task partitioning scheme for each data unit in the high-frequency data unit set.

[0122] Determination unit 602 is configured to determine the interaction frequency between each data unit based on the dependencies between subquery tasks represented by the subquery task partitioning scheme, and to determine a data distribution scheme based on the interaction frequency. The data distribution scheme indicates the storage relationship between each data unit and each node. Furthermore, based on the storage relationship indicated by the data distribution scheme, the determination unit 602 is configured to determine the index set for each node to use for querying the stored data units.

[0123] The generation unit 601 is further configured to calculate, for each index set, the execution costs of different subquery tasks based on the access patterns supported by the data units that support the query in the index set, and generate a subquery task parallelization plan based on the execution costs. The subquery task parallelization plan indicates the number of nodes required and the number of subquery tasks that each node needs to process, and to generate a node scheduling plan that satisfies the subquery task parallelization plan.

[0124] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0125] Figure 7 It is a block diagram of an electronic device 700 for query optimization of a distributed database according to an exemplary embodiment.

[0126] like Figure 7As shown, one embodiment of the present application provides an electronic device 700. The electronic device 700 includes a memory 701, a processor 702, and an input / output (I / O) interface 703. The memory 701 is used to store instructions. The processor 702 is used to call the instructions stored in the memory 701 to execute the method for query optimization of the distributed database in the embodiment of the present application. The processor 702 is connected to the memory 701 and the I / O interface 703 respectively, for example, via a bus system and / or other forms of connection mechanisms (not shown). The memory 701 can be used to store programs and data, including the program of the method for query optimization of the distributed database involved in the embodiment of the present application, and the processor 702 executes various functional applications and data processing of the electronic device 700 by running the program stored in the memory 701.

[0127] In the embodiment of the present application, the processor 702 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 702 can be a central processing unit (CPU) or one or a combination of other processing units with data processing capabilities and / or instruction execution capabilities.

[0128] The memory 701 in the embodiment of the present application may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] In the embodiment of the present application, the I / O interface 703 can be used to receive input instructions (such as digital or character information, and generate key signal input related to user settings and function control of the electronic device 700), and can also output various information to the outside (such as images or sounds). In the embodiment of the present application, the I / O interface 703 can include one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0130] In some embodiments, the present application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, any of the methods described above is performed.

[0131] In some embodiments, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it performs any of the methods described above.

[0132] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0133] The methods, systems, devices, and storage media of the present application can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement the various method steps. It should also be noted that the terms "system" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0134] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations or procedures described.

[0135] The foregoing description of the implementation of the present application has been provided for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the present application to the precise form disclosed, and various variations and modifications are possible in accordance with the above teachings or may result from the practice of the present application. These embodiments have been selected and described in order to illustrate the principles of the present application and its practical application, so as to enable those skilled in the art to utilize the present application in various embodiments and modifications as appropriate for the particular use contemplated.

[0136] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] It is further understood that, unless otherwise specified, “connection” includes a direct connection where there are no other components between the two elements, and also includes an indirect connection where there are other elements between the two elements.

[0138] It should be further understood that although operations are described in a particular order in the drawings in the embodiments of the present application, this should not be construed as requiring that these operations be performed in the particular order shown or in a serial order, or that all of the illustrated operations be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0139] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to encompass any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the field of the present application that are not disclosed herein. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the scope of claims below.

[0140] It should be understood that the present application is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the scope of the appended claims.

[0141] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A query optimization method for a distributed database, characterized in that: include: Determine a high-frequency data unit set in a distributed database, and generate a sub-query task partitioning scheme for each data unit in the high-frequency data unit set; Determining the interaction frequency between the data units based on the dependency relationship between the sub-query tasks represented by the sub-query task partitioning scheme, and determining a data distribution scheme based on the interaction frequency; wherein the data distribution scheme is used to indicate the storage relationship of the data units in each node; Determining, based on the storage relationship indicated by the data distribution scheme, an index set for each of the nodes to query the stored data units; For each of the index sets, respectively calculating execution costs of different subquery tasks based on access patterns supported by data units supported by the index set for query, and generating a subquery task parallelization plan based on the execution costs; wherein the subquery task parallelization plan is used to indicate the number of nodes required and one or more subquery tasks required to be processed by each node; Generate a node scheduling solution that satisfies the parallel solution for executing the subquery task; The step of generating a sub-query task partitioning scheme for each data unit in the high-frequency data unit set includes: Obtaining classification labels of the data units, extracting predicate relationships corresponding to access modes supported by the data units, and generating a predicate relationship set including predicate relationships and data unit identifiers through a preset predicate template matching algorithm; Extracting predicate dependency relationships from the predicate relationship set, and constructing the dependency relationship network for the predicate dependency relationships whose connection strength is higher than a preset threshold; According to the dependency network, each data unit is divided into sub-query task granularity, and an initial query set is generated according to a preset granularity control rule; For the initial query set, the access frequency of each sub-query task is sorted, and the sub-query task partitioning scheme is generated by a partitioning algorithm based on access frequency.

2. The method according to claim 1, characterized in that Determining the interaction frequency between the data units according to the dependency relationship between the sub-query tasks represented by the sub-query task division scheme includes: Obtaining data dependency relationships from the sub-query task partitioning scheme, and using the data dependency relationships as data dependency edges between the sub-query tasks to obtain a dependency graph between the sub-query tasks; According to the dependency graph, a graph partitioning algorithm is used to calculate the interaction frequency between the data units.

3. The method according to claim 1, characterized in that Determining a data distribution scheme according to the interaction frequency includes: Take each interaction frequency as a matrix element to generate an interaction frequency matrix; With respect to the interaction frequency matrix, data units corresponding to interaction frequencies higher than a preset threshold are allocated to the same node to obtain the data distribution scheme.

4. The method according to claim 1 or 3, characterized in that The method further comprises: The data distribution scheme is load-balanced and optimized for computing resource allocation using a greedy algorithm to update the data distribution scheme.

5. The method according to claim 1, characterized in that The generating of a node scheduling solution that satisfies the parallel solution for executing the subquery task includes: According to the sub-query task parallel scheme, a topological sorting algorithm is used to analyze the sub-query task dependencies, and a scheduling order is generated according to the sub-query task dependencies as an execution priority sequence of the sub-query tasks; The execution priority sequence is used as the allocation priority of the sub-query task, and the sub-query task is allocated to the corresponding node to obtain the node scheduling solution.

6. The method according to claim 1 or 5, characterized in that The method further comprises: If the node scheduling scheme satisfies the requirement that the load rate of the target node is higher than a preset load rate threshold, then calculating the resource allocation rate of the target node and adjusting the parallel granularity in combination with the node heterogeneity between the nodes to update the node scheduling scheme; and / or With respect to the node scheduling scheme, a consistent hashing algorithm is used to analyze data distribution and data locality, and the allocation relationship between the sub-query tasks and the nodes is adjusted to update the node scheduling scheme.

7. A distributed database query optimization system, characterized in that: A method for implementing a distributed database query optimization method according to any one of claims 1 to 6, comprising: A generating unit, configured to determine a high-frequency data unit set in a distributed database and generate a sub-query task partitioning scheme for each data unit in the high-frequency data unit set; a determination unit configured to determine, based on the dependencies between the sub-query tasks represented by the sub-query task partitioning scheme, an interaction frequency between the data units, and determine a data distribution scheme based on the interaction frequency; wherein the data distribution scheme indicates a storage relationship between each data unit and each node; and to determine, based on the storage relationship indicated by the data distribution scheme, an index set for each node to use for querying the stored data units; The generation unit is also used to: for each of the index sets, calculate the execution costs of different sub-query tasks respectively according to the access modes supported by the data units supported by the index set for query, and generate a sub-query task parallel plan based on the execution costs; wherein the sub-query task parallel plan is used to indicate the required number of nodes and one or more sub-query tasks required to be processed by each node; and to generate a node scheduling plan that meets the requirements of executing the sub-query task parallel plan.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images