Query optimization method and system for distributed database

Abstract

The invention provides a query optimization method and system for a distributed database. The method comprises the following steps: determining a high-frequency data unit set in a distributed database, and generating a sub-query task division scheme for each data unit in the high-frequency data unit set; according to the dependency relationship between the sub-query tasks represented by the sub-query task division scheme, determining the interaction frequency between the data units, and determining a data distribution scheme according to the interaction frequency; according to a storage relationship indicated by the data distribution scheme, determining an index set of each node for querying the stored data unit; for each index set, respectively calculating execution costs of different sub-query tasks according to an access mode supported by a query data unit supported by the index set, and generating a sub-query task parallel scheme according to the execution costs; and generating a node scheduling scheme meeting the execution sub-query task parallel scheme. The query speed of the distributed database and the overall performance of the system can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a query optimization method and system for a distributed database. Background Art

[0002] With the rapid development of information technology, distributed databases, as a core technology for modern data management, play a crucial role in processing massive data, supporting high-concurrency access, and achieving system scalability. Especially in the context of the rapid popularization of cloud computing and big data applications, the query performance of distributed databases is directly related to system efficiency and user experience. However, existing query optimization methods have shown obvious limitations in dealing with complex query requirements and highly dynamic data access patterns.

[0003] Many traditional optimization schemes rely on static partitioning or simple load balancing strategies, which are difficult to adapt to the dynamic changes in data access patterns, resulting in frequent cross-node data transmission and increased query latency. In addition, there is a lack of effective mechanisms for the coordination between data distribution optimization and query decomposition in the existing technology, which limits the improvement of the overall system performance. Especially in the field of distributed database query optimization, how to efficiently organize data distribution and optimize the query execution process is one of the core challenges. For example, uneven distribution of frequently accessed data units increases the cost of cross-node communication, and the decomposition and parallel execution of complex queries lack refined management, making it difficult to fully utilize the computing power of nodes. At the same time, the dynamic evaluation of data similarity between nodes and the low efficiency of index construction also restrict the adaptability of query optimization.

[0004] Therefore, this application provides a query optimization method and system for a distributed database to solve one of the above technical problems. Summary of the Invention

[0005] The purpose of this application is to provide a query optimization method and system for a distributed database, which can solve at least one of the above-mentioned technical problems. The specific solutions are as follows: According to the specific embodiments of this application, in the first aspect, this application provides a query optimization method for a distributed database, including: Determine a set of high-frequency data units in a distributed database, and generate a sub-query task partitioning scheme for each data unit in the set of high-frequency data units; determine the interaction frequency between the data units according to the dependency relationship between the sub-query tasks characterized by the sub-query task partitioning scheme, and determine a data distribution scheme according to the interaction frequency; wherein, the data distribution scheme is used to indicate the storage relationship of each data unit in each node; determine an index set for each node to query the stored data units according to the storage relationship indicated by the data distribution scheme; for each index set, calculate the execution cost of different sub-query tasks respectively according to the access mode supported by the data units supported by the index set, and generate a sub-query task parallelization scheme according to the execution cost; wherein, the sub-query task parallelization scheme is used to indicate the number of required nodes and one or more sub-query tasks to be processed by each node; generate a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme.

[0006] According to the specific embodiments of the present application, in a second aspect, the present application provides a query optimization system for a distributed database, including: A generating unit, configured to determine a set of high-frequency data units in a distributed database and generate a sub-query task partitioning scheme for each data unit in the set of high-frequency data units; a determining unit, configured to determine the interaction frequency between the data units according to the dependency relationship between the sub-query tasks characterized by the sub-query task partitioning scheme, and determine a data distribution scheme according to the interaction frequency; wherein, the data distribution scheme is used to indicate the storage relationship between each data unit and each node; and configured to determine an index set for each node to query the stored data units according to the storage relationship indicated by the data distribution scheme; the generating unit is further configured to: for each index set, calculate the execution cost of different sub-query tasks respectively according to the access mode supported by the data units supported by the index set, and generate a sub-query task parallelization scheme according to the execution cost; wherein, the sub-query task parallelization scheme is used to indicate the number of required nodes and one or more sub-query tasks to be processed by each node; and configured to generate a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme.

[0007] According to the specific embodiments of the present application, in a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the method described in any one of the first aspects.

[0008] According to the specific embodiments of the present application, in a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program / instructions are stored, and when the computer program / instructions are executed by a processor, the method described in any one of the first aspects is implemented.

[0009] Compared with the prior art, the above solution of the embodiment of the present application has at least the following beneficial effects: The present application provides a query optimization method for a distributed database. By determining the set of high-frequency data units in the distributed database and generating a sub-query task partitioning scheme, this method can accurately identify the data units most frequently accessed in the system, and thus optimize the storage and access methods of these data in a targeted manner. This not only improves the query efficiency but also reduces unnecessary resource consumption. Based on the dependency relationship between sub-query tasks, the interaction frequency between data units is determined, and a data distribution scheme is formulated accordingly, so that related data is stored on the same node as much as possible, greatly reducing the need for cross-node data transmission and further improving the query speed and the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Shows a flowchart of a query optimization method for a distributed database; Figure 2 Shows a flowchart of a method for generating a sub-query task partitioning scheme; Figure 3 Shows a flowchart of a method for determining the interaction frequency between data units; Figure 4 Shows a flowchart of a method for generating a sub-query task partitioning scheme; Figure 5 Shows a flowchart of a method for generating a node scheduling scheme; Figure 6 Shows a block diagram of a query optimization system for a distributed database according to an embodiment of the present application; Figure 7 Is a block diagram of an electronic device for query optimization of a distributed database shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0012] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0013] It should be understood that the term "and / or" used herein is merely a description of the associated relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this text generally represents an "or" relationship between the preceding and following associated objects.

[0014] It should be understood that although terms such as first, second, and third may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first can also be referred to as the second, and similarly, the second can also be referred to as the first.

[0015] Depending on the context, the words "if", "when" as used herein can be interpreted as "when...", "when...", "in response to determining", or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined", "in response to determining", "when detecting (stated condition or event)", or "in response to detecting (stated condition or event)".

[0016] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a commodity or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such commodity or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the commodity or system including the said element.

[0017] It should be particularly noted that symbols and / or numbers existing in the specification, if not marked in the accompanying drawings, are not reference numerals.

[0018] The optional embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0019] For the embodiments provided by the present application, that is, embodiments of a query optimization method for a distributed database.

[0020] The following will be combined with Figure 1 The embodiments of the present application will be described in detail.

[0021] Figure 1 The flowchart of a query optimization method for a distributed database is shown, as Figure 1 shown, including steps S101 to S105.

[0022] Step S101: Determine the set of high-frequency data units in the distributed database and generate a sub-query task partitioning scheme for each data unit in the set of high-frequency data units.

[0023] Step S102: Determine the interaction frequency between each data unit according to the dependency relationship between sub-query tasks characterized by the sub-query task partitioning scheme, and determine the data distribution scheme according to the interaction frequency.

[0024] Among them, the data distribution scheme is used to indicate the storage relationship of each data unit in each node.

[0025] Step S103: Determine the index set used by each node to query the stored data units according to the storage relationship indicated by the data distribution scheme.

[0026] Step S104: For each index set, calculate the execution cost of different sub-query tasks according to the access patterns supported by the data units supported by the index set, and generate a sub-query task parallelization scheme according to the execution cost.

[0027] Among them, the sub-query task parallelization scheme is used to indicate the required number of nodes and one or more sub-query tasks that each node needs to process.

[0028] Step S105: Generate a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme.

[0029] The query optimization method for the distributed database provided by this application can identify the most frequently accessed data units in the system accurately by determining the set of high-frequency data units in the distributed database and generating a sub-query task partitioning scheme, so as to optimize the storage and access methods of these data specifically. This not only improves the query efficiency, but also reduces unnecessary resource consumption. Determine the interaction frequency between data units based on the dependency relationship between sub-query tasks, and formulate a data distribution scheme accordingly, so that the relevant data is stored on the same node as much as possible, greatly reducing the need for cross-node data transmission, and further improving the query speed and the overall performance of the system.

[0030] In the embodiments of this application, the set of high-frequency data units can be determined in the following manner.

[0031] In some embodiments, extract the access frequency statistical data of each data unit from the access log of the distributed database, apply a time decay factor to reduce the access weight of historical data in the access frequency statistical data, and based on anomaly filtering processing and normalization processing, obtain the access heat value of each data unit, generate a heat distribution map for the access pattern type to which the data unit belongs, and determine the set of high-frequency data units based on the heat distribution map.

[0032] Among them, access logs are obtained from the distributed database. For example, a log parsing tool can be used to extract the access records of each data unit, and a structured access data set is generated based on a preset data unit identification rule, obtaining a log data set containing data unit identification and access timestamps.

[0033] In some embodiments, the application of the time decay factor formula can be represented by where H(t) represents the decayed access weight, C represents the original number of accesses, λ represents the time decay coefficient, t represents the time difference between the current time and the access time, and e represents the base of the natural logarithm.

[0034] In some embodiments, the access weight of historical data in the access frequency statistics data is reduced to obtain a decayed access frequency. If the decayed access frequency exceeds a preset abnormal threshold, an anomaly detection algorithm based on K-means clustering is used to filter out abnormal accesses. For the filtered access frequency, for example, the min-max normalization formula is used to determine the normalized popularity value, where N represents the normalized popularity value, with a value range between 0 and 1, F represents the current access interaction frequency value, F_min represents the minimum value among all frequencies, and F_max represents the maximum value among all frequencies. This formula can map access frequency data in different ranges to a unified interval, facilitating subsequent analysis and comparison, and obtaining the normalized access popularity value.

[0035] On this basis, according to the normalized access popularity value, a hierarchical clustering algorithm can be used to classify the access patterns of data units, and a heat map is generated through a visualization tool based on the classification results. For the data units with a popularity value higher than the preset threshold in the heat map, a set of high-frequency data units is determined. In the heat map, the distribution is based on the type of access pattern supported by the data unit, and the heat is based on the sum of the access popularity values of each data unit under a single access pattern type.

[0036] In some specific embodiments, obtaining access logs from the distributed database is the basis for analyzing the access patterns of data units. For example, an e-commerce platform stores the browsing records of users on goods, and the logs include user IDs, product IDs, and timestamps, etc. The log parsing tool can use regular expressions to extract these fields and generate a structured data set.

[0037] For example, each row of the parsed dataset is in the form: the product ID is P001, and the access time is 2025-04-23 10:00:00. Compared with manual parsing, the automated tool significantly improves efficiency and ensures data consistency. A structured dataset is generated based on a preset data unit identification rule, which usually takes a unique identifier such as the product ID as the core, that is, the product ID is used as the data unit identifier. For example, product P001 represents a data unit, where P001 represents the data unit identifier of this data unit.

[0038] In a feasible implementation, the rule can be defined as starting with the letter P followed by three digits to filter out invalid records. For example, P001 and P002 are valid product IDs, and after parsing, a log dataset containing product IDs and timestamps is formed. This rule-based design facilitates subsequent statistics and analysis and reduces data redundancy. For the log dataset, a counting algorithm counts the access frequency of each data unit. Specifically, P001 was accessed 100 times within 24 hours, and P002 was accessed 50 times. The time decay factor formula is applied to reduce the access weight of historical data in the access frequency statistics.

[0039] Preferably, the decay coefficient λ is set to 0.1, and the time difference t is in hours. Older access records have reduced weight, reflecting recent access trends. This method makes the access frequency more closely aligned with real-time user interests and improves the timeliness of analysis. If the decayed access frequency exceeds an abnormal threshold, an anomaly detection algorithm based on K-means clustering can identify abnormal accesses. For example, the decayed frequency of P001 is 80, exceeding the threshold of 50, which may be due to a promotion causing a surge in accesses. K-means clustering classifies access patterns into normal and abnormal categories, and after filtering out the anomalies, the real user behavior data is retained. This filtering mechanism effectively eliminates the influence of crawlers or malicious accesses and ensures data reliability. For the filtered access frequency, the min-max normalization formula maps the frequency to the range of 0 to 1.

[0040] In some embodiments, the frequency of P001 is 80, P002 is 40, the minimum frequency F_min is 40, the maximum frequency F_max is 80. After normalization, the popularity value of P001 is 1, and P002 is 0. This normalization facilitates comparing the relative popularity of different data units, eliminates the dimension difference, and improves the visualization effect. According to the normalized access popularity value, a hierarchical clustering algorithm classifies the access patterns of data units. It can be understood that the algorithm groups products into high-popularity, low-popularity, etc. groups according to the similarity of popularity values. For example, P001 and P003 are classified into the high-popularity group, reflecting similar user preferences. The classification results generate a popularity distribution map through a visualization tool, and the hot spots intuitively show the products with high-frequency accesses. This visualization helps the operation team quickly identify popular products and optimize the recommendation strategy.

[0041] Exemplarily, for data units in the heat distribution map with heat values higher than a preset threshold, a high-frequency data unit set is determined. For example, P001 and P003 with heat values greater than 0.8 form a high-frequency set. These products can be preferentially used for marketing promotion, significantly improving the conversion rate. It should be noted that the dynamic update of the high-frequency set helps to adjust the inventory and advertising placement strategies in real time, enhancing the platform operation efficiency.

[0042] In the embodiments of the present application, a sub-query task partitioning scheme can be generated in the following manner.

[0043] Figure 2 A method flow chart for generating a sub-query task partitioning scheme is shown, as Figure 2 shown, including steps S201 to S204.

[0044] Step S201: Obtain the classification labels of each data unit, extract the predicate relationships corresponding to the access modes supported by each data unit, and generate a predicate relationship set containing the predicate relationships and data unit identifiers through a preset predicate template matching algorithm.

[0045] Step S202: Extract the predicate dependency relationships in the predicate relationship set, and construct a dependency relationship network for the predicate dependency relationships with connection strengths higher than a preset threshold.

[0046] Step S203: According to the dependency relationship network, perform sub-query task granularity partitioning on each data unit, and generate an initial query set through a preset granularity control rule.

[0047] Step S204: For the initial query set, sort the access frequencies of each sub-query task, and generate a sub-query task partitioning scheme through a partitioning algorithm based on the access frequency.

[0048] In the embodiments of the present application, the classification labels of high-frequency data units are obtained from the heat distribution map, the predicate relationships corresponding to the access modes of the data units are extracted using a logical parsing tool, and a structured relationship data set containing the predicate relationships and data unit identifiers is generated through a preset predicate template matching algorithm to obtain a predicate relationship set. Among them, for the predicate relationship set, the dependency relationships between the predicate relationships are extracted using a graph parsing algorithm. If the connection strength of the dependency relationship is higher than a preset threshold, a dependency relationship network is constructed through a weighted directed graph generation tool to obtain a dependency relationship network. According to the dependency relationship network, the K-means clustering algorithm is used to perform sub-query task granularity partitioning on the data units, and an initial query set is generated through a preset granularity control rule. In addition, for the initial query set, an access frequency statistical data tool is used to sort the access frequencies of each sub-query task, and a sub-query task partitioning scheme is generated through a partitioning algorithm based on the access frequency to determine the sub-query task partitioning scheme.

[0049] In the above embodiments, when constructing the sub-query task partitioning scheme, by obtaining the classification labels of each data unit and extracting the predicate relationships corresponding to the access patterns it supports, and using the predicate template matching algorithm to generate a set of predicate relationships, this method can effectively capture the internal connections between user behavior patterns. Further, after establishing the dependency relationship network, the sub-query task granularity is partitioned according to the dependency strength, ensuring that each sub-query task can cover the key aspects of user behavior. At the same time, the query partitioning is guided by the result of sorting based on the access frequency, and the high-frequency tasks are processed preferentially, which can significantly improve the system response speed and user experience.

[0050] In some specific embodiments, the classification labels of high-frequency data units are obtained from the heat distribution map, and the predicate relationships corresponding to the access patterns can be extracted through a logical parsing tool. For example, in an e-commerce platform, the heat distribution map shows that products P001 and P003 are high-frequency data units, and the classification label is "high heat". The logical parsing tool analyzes the user access logs and extracts predicate relationships such as "user U001 views P001".

[0051] In some specific embodiments, the tool generates triples in the form of "subject-predicate-object" based on the user behavior in the logs. The predicate relationships of P001 may include "view", "favorite", etc. This method is convenient for structurally describing the interaction between users and products.

[0052] As a feasible implementation, the preset predicate template matching algorithm generates a structured relationship data set as the set of predicate relationships.

[0053] For example, the template is defined as "user ID-behavior-product ID", which matches the records in the logs, such as "U001-view-P001". The algorithm scans the logs and extracts all records that match the template to form a data set containing predicate relationships and product IDs. Preferably, the data set is grouped by product ID to facilitate subsequent analysis of the distribution of predicate relationships. For the set of predicate relationships, the graph parsing algorithm extracts the dependency relationships.

[0054] For example, there is a dependency between "view" and "favorite" of P001. If the frequency of users viewing and then favoriting is high, the dependency relationship strength is high. Further, the algorithm counts the number of occurrences of the "view-favorite" sequence and calculates the strength value. For example, the strength of P001 is 0.8, which is higher than the threshold of 0.6.

[0055] In the embodiments of the present application, the high-strength dependency relationships reflect user behavior patterns, which are helpful for accurate recommendation. A dependency relationship network can be constructed through a weighted directed graph generation tool. It should be understood that the nodes are predicates such as "browse" and "favorite", the edges are dependency relationships, and the weights are strength values. For example, the network of P001 shows that "browse" points to "favorite" with a weight of 0.8. Such a network intuitively displays the associations between behaviors and is convenient for analyzing complex patterns. According to the dependency relationship network, the K-means clustering algorithm divides the sub-query task granularity. For example, the predicate relationship patterns of P001 and P003 are similar and are clustered into the same sub-query task group.

[0056] Preferably, the granularity control rule sets that each sub-query task contains at least two predicate relationships to ensure that the query covers diverse behaviors. For example, the initial query set is in the form of "browse and favorite the records of P001".

[0057] As a feasible implementation, the access frequency statistical data tool sorts the sub-query tasks. For example, the query frequency of "browse and favorite P001" is 100 times, and that of "only browse P002" is 50 times. Based on the division algorithm of access frequency, a sub-query task division scheme is generated, and high-frequency queries are preferentially retained, such as the queries related to P001. This scheme focuses on the access patterns of popular products and is helpful for optimizing recommendation and inventory management.

[0058] Figure 3 A method flow chart for determining the interaction frequency between data units is shown, as Figure 3 shown, including steps S301 to S302.

[0059] Step S301, obtain the data dependency relationships from the sub-query task division scheme, and use the data dependency relationships as the data dependency edges between sub-query tasks to obtain the dependency relationship graph between sub-query tasks.

[0060] Step S302, according to the dependency relationship graph, use the graph segmentation algorithm to calculate the interaction frequency between data units.

[0061] In the embodiments of the present application, extracting the data dependency relationships from the sub-query task division scheme, converting them into a dependency relationship graph, and using the graph segmentation algorithm to calculate the interaction frequency between data units. This method provides a scientific basis for subsequent data distribution optimization. It not only helps to identify which data units are closely related, but also helps to understand the data flow of the entire system, laying a foundation for reducing unnecessary data transmission and improving query efficiency.

[0062] Figure 4 A method flow chart for generating a sub-query task division scheme is shown, as Figure 2 shown, including steps S401 to S402.

[0063] Step S401: Generate an interaction frequency matrix by taking each interaction frequency as a matrix element.

[0064] Step S402: For the interaction frequency matrix, allocate the data units corresponding to the interaction frequencies higher than a preset threshold to the same node to obtain a data distribution scheme.

[0065] In the embodiments of the present application, generating an interaction frequency matrix by taking each interaction frequency as a matrix element and allocating the data units corresponding to the interaction frequencies higher than a preset threshold to the same node effectively realizes the optimization of data locality and reduces the cost of cross-node communication. For data units with frequent interactions, such a layout greatly improves the query efficiency, simplifies the system design, and enhances the scalability and flexibility of the system.

[0066] In some specific embodiments, in the recommendation system of an e-commerce platform, extracting data dependency relationships from the sub-query task partitioning scheme is the key to optimizing data processing. Data dependency relationships reflect the data interaction patterns between sub-query tasks, such as the associations between user behavior queries. The association analysis algorithm scans the results of sub-query tasks, extracts frequent item sets, and generates data dependency edges.

[0067] Exemplarily, sub-query task Q1 is "the record of user browsing product P001", and Q2 is "the record of user collecting P001". The algorithm analyzes the log and finds that the output data of Q1 and Q2 are often used by subsequent recommendation modules simultaneously, generating a dependency edge Q1→Q2 with a weight of the co-occurrence times, such as 80 times, reflecting a strong association. The dependency relationship graph between sub-query tasks takes sub-query tasks as nodes and dependency edges as directed edges, intuitively showing the data flow direction. Based on the dependency relationship graph between sub-query tasks, the graph partitioning algorithm calculates the interaction frequency values between data units. Data units refer to entities such as products or users, and the interaction frequency value refers to the co-occurrence times of entities in sub-query tasks.

[0068] In some specific embodiments, the algorithm traverses the dependency graph, counts the co-occurrence times of P001 and P003 in Q1 and Q2, such as the co-occurrence times of P001 and P003 being 50 times, and generates an interaction frequency matrix. The matrix elements are interaction frequency values, such as P001-P003 being 50 and P001-P002 being 20. If the interaction frequency value is higher than the threshold of 30, it is considered that the association strength is high.

[0069] Preferably, P001 and P003 are allocated to the same computing node to generate an initial data distribution scheme, ensuring that high-frequency interaction data is processed on the same node and reducing cross-node communication.

[0070] As a feasible implementation, the load balancing adjusts and optimizes the initial data distribution scheme. The greedy algorithm evaluates the loads of each computing node. For example, node A processes P001 and P003 with a load of 100 queries per second, and node B processes P002 with a load of 50 queries per second. The algorithm migrates some low-frequency data, such as P002, to the node with a lower load to generate an updated data distribution scheme. It should be noted that when making adjustments, the storage capacity of the nodes is considered. For example, the capacity of node A is 500GB, which is sufficient to accommodate the data of P001 and P003. The updated data distribution scheme ensures load balancing and consistent query efficiency among nodes. It can be understood that the threshold judgment of the interaction frequency matrix facilitates the identification of key data associations, and allocating them to the same node reduces query latency. The dynamic adjustment of the greedy algorithm adapts to changes in the data volume and ensures the stability of the system. For example, the high-frequency queries of P001 are allocated to high-performance nodes to prioritize the response to user requests. This method optimizes resource allocation by refining data dependency and interaction analysis, improving the response speed and accuracy of the recommendation system.

[0071] In the embodiments of the present application, the following method can be used to determine the index set for each node to query the stored data unit.

[0072] In some embodiments, according to the access pattern and classification rules, a pre-established clustering algorithm can be used to classify data requests, generate a hierarchical index framework, and determine the hierarchical index structure of the access pattern. Through data granularity and storage allocation, index entries are extracted from the hierarchical index framework, and a hash table is used to construct a set of index entries local to the node to obtain the index set.

[0073] For example, the processing of the access pattern and classification rules can be achieved by analyzing the characteristics of data requests. The access pattern refers to the query habits of users or systems for data, such as frequently accessing certain fields or querying within a time range. The classification rules are predefined based on dimensions such as the frequency of requests, data types, or access times. A clustering algorithm, such as K-means clustering, can group requests according to characteristics. For example, in a distributed database, assume there are 10 million request logs containing user IDs, query fields, and timestamps. After analysis by the clustering algorithm, the requests are divided into a high-frequency field query group and a low-frequency range query group, generating a hierarchical index framework. This framework organizes the indexes in a tree structure, with the top layer being the high-frequency access pattern and the bottom layer being the low-frequency pattern, ensuring fast data location.

[0074] In some specific embodiments, the construction of the hierarchical index structure depends on the clustering results. For example, fields in the high-frequency query group, such as "order amount", are assigned to the top-level nodes of the index tree, and the nodes store pointers to the data. Low-frequency queries, such as "historical remarks", are placed at the bottom layer. This structure facilitates the rapid retrieval of high-frequency data. Suppose that among 100,000 daily queries, 80% are concentrated on the order amount field, and the top-level index can reduce the retrieval time by 90%. It should be noted that the hierarchical index framework needs to be updated regularly to adapt to changes in the access pattern, such as reclustering once a month.

[0075] As a feasible implementation, when extracting index entries from the hierarchical index framework, the data granularity needs to be considered. Data granularity refers to the shard size of the data, such as dividing by day or by user ID. Suppose the data is sharded by day, and each shard contains 100,000 records. The index entry records the storage location of each shard of data. A hash table is used to construct the set of index entries local to the node because of its high search efficiency. For example, a certain node manages 10 data shards. The hash table uses the shard ID as the key, and the value is the index entry. When querying the shard ID "2023-04-01", it can be directly located.

[0076] Preferably, the hash table needs to set an appropriate bucket size, such as 1000 buckets, to reduce the collision rate.

[0077] It can be understood that the generation of the index set needs to balance storage and query efficiency. For example, a certain node stores 1000 index entries, and each entry contains a shard ID and a pointer, occupying about 10KB of memory. The average query time of the hash table is 1 microsecond, which is suitable for high-concurrency scenarios. In another embodiment, redundant index entries can be added for hot data, such as storing an additional index for the frequently accessed order amount field, pointing to the shard replicas of different nodes, to improve fault tolerance. This method optimizes from multiple aspects, such as hot data priority, redundant indexes, and efficient hash table queries, to jointly support the goal of fast access.

[0078] As a feasible implementation, for dynamic access patterns, an adaptive adjustment mechanism can be introduced. Suppose the query frequency of a certain field suddenly increases from low frequency to high frequency. After the system detects it, its index entry will be promoted to the top layer. This mechanism ensures that the index framework always matches the actual needs through the collaborative work of the clustering algorithm and the hash table. It should be noted that the adaptive adjustment needs to control the frequency, such as adjusting once a day, to avoid resource waste caused by frequent index reconstruction. This method progresses step by step from access pattern analysis to index construction, forming an efficient data access system.

[0079] In some other embodiments, the data distribution scheme can be further adjusted by considering the load balance. For example, perform load balancing adjustment on the data distribution scheme and use a greedy algorithm to optimize the computing resource allocation to update the data distribution scheme. In the above embodiments, the load balancing adjustment ensures that the workloads of all nodes are relatively balanced, avoiding the situation of individual nodes being overloaded. It can not only make full use of system resources but also ensure that the system can still operate efficiently and stably in high-concurrency scenarios, improving the reliability and stability of the system.

[0080] In the embodiments of the present application, the node scheduling scheme can be generated in the following manner.

[0081] Figure 5 A method flowchart for generating a node scheduling scheme is shown, as Figure 5 shown, including step S501 to step S502.

[0082] Step S501: According to the subquery task parallelism scheme, use the topological sorting algorithm to analyze the dependencies of subquery tasks, and generate a scheduling order according to the dependencies of subquery tasks as the execution priority sequence of subquery tasks.

[0083] Step S502: Use the execution priority sequence as the allocation priority of subquery tasks, and allocate subquery tasks to corresponding nodes to obtain the node scheduling scheme.

[0084] In the embodiments of the present application, using the topological sorting algorithm to analyze the dependencies of subquery tasks and generate a scheduling order enables the subquery tasks to be executed in the correct dependency order, preventing data inconsistency or query failure problems caused by incorrect order. This way ensures that each subquery task is executed at an appropriate time, maximizing the use of system resources and improving query efficiency and accuracy.

[0085] In the embodiments of the present application, the node scheduling scheme can also be updated by one or a combination of the following methods.

[0086] Method 1: If the node scheduling scheme satisfies that the load ratio of the target node is higher than the preset load ratio threshold, calculate the resource allocation ratio of the target node, and adjust the parallelism granularity in combination with the node heterogeneity between nodes to update the node scheduling scheme. Herein, the target node refers to one or several nodes that meet the above threshold conditions among all nodes.

[0087] Method 2: For the node scheduling scheme, use the consistent hashing algorithm to analyze the data distribution and data locality, and adjust the allocation relationship between subquery tasks and nodes to update the node scheduling scheme.

[0088] In the embodiments of the present application, when the node scheduling scheme may lead to too high a load rate of some nodes, the parallel granularity is adjusted by calculating the resource allocation rate of the target node and combining node heterogeneity. This method can dynamically adjust the task allocation strategy without affecting the overall performance of the system, achieving more reasonable resource allocation. In addition, the consistent hashing algorithm is used to analyze data distribution and data locality, further optimizing the allocation relationship between sub-query tasks and nodes, ensuring the efficiency of data access and the system's scalability. The combined effect of these multi-level optimization measures enables the system to perform well in the face of complex queries and highly dynamic data access patterns.

[0089] For example, in a distributed database, when analyzing the dependencies of the sub-query task parallelism scheme, the topological sorting algorithm can be used to generate an ordered preliminary scheduling order. Topological sorting identifies the dependency relationships between sub-query tasks by constructing a directed acyclic graph. For example, a certain order query involves an order table and a user table, where the order status query depends on the user authentication query. The system models these two sub-query tasks as graph nodes and the dependency relationship as an edge. Topological sorting generates the order: first execute the user authentication, and then execute the order status query. This way ensures that sub-query tasks are executed in the order of dependencies, avoiding data inconsistency.

[0090] As a feasible implementation, based on the execution priority sequence generated by topological sorting, the system checks whether the node load rate is higher than a preset threshold, such as 80%. If the load rate of a certain node reaches 90%, the resource allocation rate is calculated through a hash table. The hash table uses the node ID as the key, and the values are the CPU occupancy rate and memory usage. For example, the CPU occupancy rate of node A is 85% and the memory occupancy is 70%. Combining node heterogeneity, such as node A having an 8-core CPU and node B having a 16-core CPU, the system adjusts the parallel granularity, allocating computationally intensive sub-query tasks to node B to reduce the pressure on node A.

[0091] In some specific embodiments, the optimized node scheduling scheme needs to consider data distribution and data locality. The consistent hashing algorithm can be used to allocate sub-query tasks to the corresponding nodes. For example, order data is sharded and stored on nodes A and B. The system calculates the hash value of the data key through consistent hashing and maps it to node A. Sub-query tasks involving the query of order status are preferentially allocated to node A, reducing cross-node data transmission. This method utilizes data locality to reduce network overhead.

[0092] Preferably, the consistent hashing algorithm also supports dynamic adjustment. If a new node C is added, the system recalculates the hash ring, and some data keys are mapped to node C, and the sub-query tasks are reallocated accordingly. For example, after adding a new node, some sub-query tasks of the order status query are allocated to node C to balance the load. This way adapts to node changes and ensures the scalability of the system.

[0093] As a feasible implementation, the generated locally optimized node scheduling scheme further refines the execution details. For example, when node A processes order status queries, it preferentially uses the local index to reduce disk I / O. The system dynamically adjusts the memory allocation of subquery tasks according to the execution plan, such as allocating an additional 2MB cache for high-priority subquery tasks. This refined management improves the query efficiency.

[0094] Among them, it can be understood that the above method forms a complete execution optimization chain from dependency analysis to task allocation. Topological sorting ensures dependency correctness, and the hash table and consistent hashing algorithm cooperate to optimize resource allocation and data locality. The local optimization plan further improves the execution efficiency. This multi-level cooperation scheme supports high-concurrency query scenarios and adapts to the dynamic changes of the distributed environment.

[0095] In the embodiments of the present application, after generating the node scheduling scheme, index optimization can be further implemented according to the node scheduling scheme.

[0096] For example, in some embodiments, the data interaction requirements between nodes can be extracted from the node scheduling scheme, and the kernel function algorithm is used to calculate the similarity matrix of data units. If the similarity value is higher than the preset threshold, it is marked as high-similarity data, and a similarity grouping based on data granularity division is obtained. Further, according to the similarity grouping, the local index storage of the node can be updated by dynamic index adjustment. If the index access efficiency is higher than the preset threshold, an optimized index set adapted to high concurrency is generated.

[0097] In the above embodiments, for example, the data interaction requirements between nodes can be extracted from the node scheduling scheme, and a graph database can be used to store the interaction relationships between nodes. By querying the edge weights between nodes in the graph database, a set of data interaction requirements between nodes can be obtained. According to the set of data interaction requirements between nodes, the kernel function algorithm is used to calculate the feature vectors of each data unit, and a similarity matrix is generated. If the element value in the similarity matrix is higher than the preset threshold, it is marked as a highly similar data unit, and a set of highly similar data is obtained. Through the set of highly similar data, based on the data granularity division rules, a clustering algorithm is used to group the highly similar data units to obtain a set of similarity groups. According to the data characteristics, a clustering algorithm is used to group the data by similarity, obtain the grouped data set, and determine the feature vectors of the data set. For the grouped data set, a B+ tree structure is used to generate a local index locally at the node, obtain a set of local indexes, and determine the storage format of the set of local indexes. According to the access pattern, the access frequency and distribution are obtained from the query log, and a dynamic adjustment algorithm is used to update the set of local indexes to determine whether the access efficiency of the set of local indexes is higher than the preset threshold, and an adjusted set of indexes is obtained. If the access efficiency of the adjusted set of indexes is higher than the preset threshold, a hash table is used to generate an optimized set of indexes, obtain an index structure suitable for high concurrency, and determine the concurrency adaptation ability of the optimized set of indexes.

[0098] In some specific embodiments, in a distributed database, the analysis of the data interaction requirements between nodes in the node scheduling scheme is the key to optimizing query efficiency.

[0099] Exemplarily, the data interaction requirements between nodes can be modeled using a graph database. In the graph database, nodes represent computing nodes, edges represent data interactions, and edge weights record interaction frequency values or data volumes. For example, an order query involves node A and node B. Node A processes user data, and node B processes order data. The system counts the interaction frequency value and finds that the frequency of node A transmitting user identity data to node B is 100 times per second, and the edge weight is set to 100. Querying the edge weights in the graph database generates a set of data interaction requirements, including the 100 times / second interaction record from node A to node B. This method clearly maps the interaction relationships and facilitates subsequent optimization.

[0100] In some specific embodiments, based on the set of data interaction requirements, the kernel function algorithm is used to calculate the feature vectors of data units. The kernel function generates feature vectors representing the behavioral characteristics of data units by analyzing the interaction patterns of data units. For example, the feature vectors of order data include the interaction frequency values of order ID, timestamp, and user ID. The system calculates a similarity matrix for the feature vectors, and the matrix element values represent the similarity between data units. If the preset threshold is 0.8, the data units with element values higher than 0.8 in the matrix are marked as highly similar. For example, the data units with order IDs 1001 and 1002 are marked as highly similar data units with a similarity of 0.9 because their timestamps are close and their user IDs are the same, forming a highly similar data set. This method effectively identifies the correlation between data.

[0101] Preferably, the highly similar data set is grouped by data granularity through a clustering algorithm. The clustering algorithm groups similar data units into a group according to the feature vectors of data units. For example, the K-means clustering algorithm divides order data into three groups: recent orders, historical orders, and high-frequency user orders based on the Euclidean distance of feature vectors. Each group has a different data granularity. For example, the recent order group contains 1000 records and has a finer data granularity. After grouping, a similarity grouping set is generated, which contains the information of the three groups of order data. This grouping method facilitates targeted query optimization.

[0102] As a feasible implementation, the edge weight query of the graph database supports dynamic update. For example, after adding a new node C, the system re-statistics the interaction frequency value from node A to node C and updates the edge weight to 50. The kernel function algorithm also adjusts the feature vector calculation logic accordingly, incorporating the data interaction pattern of node C. The clustering algorithm then re-groups according to the updated highly similar data set to ensure that the grouping results adapt to system changes. This dynamic adjustment mechanism improves the scalability of the system. It can be understood that each link of the above method supports each other. The graph database clearly stores the interaction relationships, the kernel function algorithm accurately extracts data features, and the clustering algorithm optimizes data grouping.

[0103] For example, in an order query scenario, the system identifies high-frequency interaction nodes through the graph database. After the kernel function algorithm generates an accurate similarity matrix and the clustering algorithm groups the order data, the query can preferentially process the high-frequency user order group, reducing unnecessary data scans. This multi-level collaborative optimization significantly improves the query efficiency and adapts to the dynamics of the distributed environment at the same time.

[0104] For example, in a distributed database system, a clustering algorithm based on data characteristics can be used for similarity grouping. Suppose a distributed database stores user behavior data of an e-commerce platform, and the data characteristics include user purchase frequency, browsing duration, and product category preference. Clustering algorithms such as K-Means can group users according to these characteristics. For example, users with high purchase frequency and a preference for electronic products can be grouped together. After grouping, a feature vector for each group is generated, reflecting the commonalities of the users within the group. For example, the feature vector of a certain group may show that 90% of the users prefer mobile phone products and make an average of 3 purchases per month. This grouping facilitates subsequent index optimization and query acceleration.

[0105] As a feasible implementation, for the grouped data set, a local index in the form of a B+ tree is generated locally at the node. The B+ tree is suitable for the distributed environment due to its efficient range query and sequential access characteristics. For example, a node stores the data of the above-mentioned high-frequency purchase user group and can build a B+ tree index based on the user ID or purchase time. The storage format of the local index set needs to consider the node storage capacity and query requirements. Preferably, the non-leaf nodes are stored in a compressed format to reduce the I / O overhead. Suppose a node stores 1 million records, and the B+ tree index can reduce the query time from seconds for a full table scan to milliseconds.

[0106] In some specific embodiments, analyzing the access pattern based on the query log is the key to optimizing the index. The query log records the frequency and distribution of user queries. For example, it is found that 60% of the queries are concentrated on the order data in the most recent week and are mostly filtered by time range. The dynamic adjustment algorithm can update the local index set accordingly. For example, more index space is preferentially allocated to the data in the time period with high access frequency. During the adjustment process, it is necessary to determine whether the access efficiency is higher than a preset threshold. For example, it is required that the average query latency is lower than 50 milliseconds. If the query latency of a certain node is reduced from 80 milliseconds to 40 milliseconds after the index adjustment, it meets the threshold requirement.

[0107] As a feasible implementation, if the access efficiency of the adjusted index set meets the standard, a hash table can be further used to generate an optimized index set to adapt to the high-concurrency scenario. The hash table is suitable for exact-match queries. For example, it can quickly locate order records by the user ID. Suppose the system needs to support 100,000 concurrent queries per second, and the hash table can control the single-query latency within the microsecond level. The concurrent adaptation ability of the optimized index set is reflected in its low collision rate and high throughput. For example, by dynamically expanding the hash bucket, it is ensured that 99% of the queries have no collisions.

[0108] It should be noted that the combination of hash table and B+ tree can take into account both range query and point query, significantly improving the overall performance of the system. It can be understood that the above solution ensures the efficient query capability of distributed database in high concurrency scenarios through the progressive logic from clustering grouping to index optimization. The implementation of each technical theme revolves around a single scenario of e-commerce user behavior data, ensuring the focus and practicality of the solution.

[0109] The present application also provides a system embodiment that is based on the above embodiment, which is used to implement the method steps of the above embodiment. The explanation based on the same name meaning is the same as the above embodiment, and has the same technical effect as the above embodiment, which will not be repeated here.

[0110] like Figure 6 As shown, the present application provides a distributed database query optimization system 600, including: The generating unit 601 is used to determine a high-frequency data unit set in a distributed database, and generate a sub-query task division scheme for each data unit in the high-frequency data unit set.

[0111] The determination unit 602 is used to determine the interaction frequency between each data unit according to the dependency relationship between sub-query tasks represented by the sub-query task division scheme, and determine the data distribution scheme according to the interaction frequency. The data distribution scheme is used to indicate the storage relationship between each data unit and each node. And it is used to determine the index set for each node to query the stored data unit according to the storage relationship indicated by the data distribution scheme.

[0112] The generation unit 601 is also used to: for each index set, according to the access mode supported by the data unit supported by the index set, respectively calculate the execution cost of different sub-query tasks, and generate a sub-query task parallel plan according to the execution cost. The sub-query task parallel plan is used to indicate the required number of nodes and one or more sub-query tasks to be processed by each node. And it is used to generate a node scheduling plan that satisfies the execution of the sub-query task parallel plan.

[0113] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] Figure 7 It is a block diagram of an electronic device 700 for query optimization of a distributed database according to an exemplary embodiment.

[0115] like Figure 7As shown, an embodiment of the present application provides an electronic device 700. Among them, the electronic device 700 includes a memory 701, a processor 702, and an input / output (I / O) interface 703. Among them, the memory 701 is used to store instructions. The processor 702 is used to call the instructions stored in the memory 701 to execute the method for query optimization of the distributed database in the embodiments of the present application. Among them, the processor 702 is respectively connected to the memory 701 and the I / O interface 703, and can be connected through a bus system and / or other forms of connection mechanisms (not shown) for example. The memory 701 can be used to store programs and data, including the program for the method for query optimization of the distributed database involved in the embodiments of the present application. The processor 702 executes various functional applications and data processing of the electronic device 700 by running the program stored in the memory 701.

[0116] In the embodiments of the present application, the processor 702 can be implemented in at least one of the hardware forms of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 702 can be a central processing unit (CPU) or a combination of one or several of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0117] The memory 701 in the embodiments of the present application may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), etc.

[0118] In the embodiments of the present application, the I / O interface 703 can be used to receive input instructions (such as numerical or character information, and key signal inputs related to user settings and function controls of the electronic device 700, etc.), and can also output various information to the outside (such as images or sounds, etc.). In the embodiments of the present application, the I / O interface 703 can include one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel, etc.

[0119] In some embodiments, the present application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, perform any of the methods described above.

[0120] In some embodiments, the present application provides a computer program product including a computer program that, when executed by a processor, performs any of the methods described above.

[0121] Although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to obtain the desired result. Multitasking and parallel processing may be advantageous in certain environments.

[0122] The methods, systems, devices, and storage media of the present application can be accomplished using standard programming techniques, implementing various method steps using rule-based logic or other logics. It should also be noted that the terms "system" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving inputs.

[0123] Any of the steps, operations, or programs described herein can be executed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or programs.

[0124] For purposes of illustration and description, the foregoing description of the embodiments of the present application has been given. The foregoing description is not exhaustive nor is it intended to limit the present application to the exact form disclosed, and various modifications and variations are possible in light of the above teachings, or may be derived from practice of the present application. These embodiments were chosen and described in order to explain the principles of the present application and its practical application so that those skilled in the art can utilize the present application in various embodiments and various modifications suitable for the particular purposes contemplated.

[0125] Regarding the system in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0126] It can be further understood that, unless otherwise specified, "connection" includes both direct connection without other components between the two and indirect connection with other elements between the two.

[0127] It can be further understood that although the operations are described in a specific order in the drawings in the embodiments of the present application, it should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all the operations shown to obtain the desired result. In a specific environment, multitasking and parallel processing may be advantageous.

[0128] Those skilled in the art will readily think of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include well-known common general knowledge or conventional technical means in the technical field of the present application that are not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0129] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

[0130] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A query optimization method for a distributed database, characterized in that Including: Determine a set of high-frequency data units in a distributed database, and generate a sub-query task partitioning scheme for each data unit in the set of high-frequency data units; According to the dependency relationship between sub-query tasks characterized by the sub-query task partitioning scheme, determine the interaction frequency between the data units, and determine a data distribution scheme according to the interaction frequency; wherein, the data distribution scheme is used to indicate the storage relationship of each data unit in each node; According to the storage relationship indicated by the data distribution scheme, determine an index set for each node to query the stored data units; For each index set, calculate the execution cost of different sub-query tasks respectively according to the access mode supported by the data units supported by the index set, and generate a sub-query task parallelization scheme according to the execution cost; wherein, the sub-query task parallelization scheme is used to indicate the number of required nodes, and one or more sub-query tasks to be processed by each node; Generate a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme.

2. The method according to claim 1, wherein The generating a sub-query task partitioning scheme for each data unit in the set of high-frequency data units includes: Obtain the classification labels of the data units, extract the predicate relationships corresponding to the access modes supported by the data units, and generate a set of predicate relationships including predicate relationships and data unit identifiers through a preset predicate template matching algorithm; Extract predicate dependency relationships in the set of predicate relationships, and construct a dependency relationship network for the predicate dependency relationships with a connection strength higher than a preset threshold; According to the dependency relationship network, perform sub-query task granularity partitioning on the data units, and generate an initial query set through a preset granularity control rule; For the initial query set, sort the access frequencies of each sub-query task, and generate the sub-query task partitioning scheme through a partitioning algorithm based on access frequency.

3. The method according to claim 1, characterized in that The determining the interaction frequency between the data units according to the dependency relationship between sub-query tasks characterized by the sub-query task partitioning scheme includes: Obtain data dependency relationships from the sub-query task partitioning scheme, and use the data dependency relationships as data dependency edges between sub-query tasks to obtain a dependency relationship graph between sub-query tasks; According to the dependency relationship graph, use a graph partitioning algorithm to calculate the interaction frequency between the data units.

4. The method according to claim 1, characterized in that, The determining the data distribution scheme according to the interaction frequency includes: Use each interaction frequency as a matrix element to generate an interaction frequency matrix; For the interaction frequency matrix, allocate the data units corresponding to the interaction frequencies higher than a preset threshold to the same node to obtain the data distribution scheme.

5. The method according to claim 1 or 4, characterized in that, The method further includes: Perform load balancing adjustment on the data distribution scheme, and use a greedy algorithm to optimize the calculation resource allocation to update the data distribution scheme.

6. The method according to claim 1, wherein The generating a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme includes: According to the sub-query task parallelization scheme, use a topological sorting algorithm to analyze the dependency of the sub-query tasks, and generate a scheduling order according to the dependency of the sub-query tasks as an execution priority sequence of the sub-query tasks; Using the execution priority sequence as the allocation priority of the sub-query tasks, allocate the sub-query tasks to the corresponding nodes to obtain the node scheduling scheme.

7. The method according to claim 1 or 6, characterized in that, The method further includes: If the node scheduling scheme satisfies that the load ratio of the target node is higher than the preset load ratio threshold, calculate the resource allocation ratio of the target node, and adjust the parallelism granularity in combination with the node heterogeneity among the nodes to update the node scheduling scheme; and / or For the node scheduling scheme, use the consistent hashing algorithm to analyze the data distribution and data locality, and adjust the allocation relationship between the sub-query tasks and the nodes to update the node scheduling scheme.

8. A query optimization system for a distributed database, characterized in that, It includes: A generation unit, configured to determine a set of high-frequency data units in a distributed database and generate a sub-query task partitioning scheme for each data unit in the set of high-frequency data units; A determination unit, configured to determine the interaction frequency among the data units according to the dependency relationship among the sub-query tasks characterized by the sub-query task partitioning scheme, and determine the data distribution scheme according to the interaction frequency; wherein, the data distribution scheme is used to indicate the storage relationship between each data unit and each node; and is configured to determine an index set for each node to query the stored data units according to the storage relationship indicated by the data distribution scheme; The generation unit is further configured to: for each index set, calculate the execution costs of different sub-query tasks respectively according to the access modes supported by the data units supported by the index set, and generate a sub-query task parallelization scheme according to the execution costs; wherein, the sub-query task parallelization scheme is used to indicate the required number of nodes and one or more sub-query tasks to be processed by each node; and is configured to generate a node scheduling scheme that satisfies the execution of the sub-query task parallelization scheme.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the method according to any one of claims 1-7 is implemented.

Images