A distributed database query method, system, medium and program product

Abstract

A distributed database query method, system, medium, and program product relate to the field of electronic digital data processing. The method comprises: parsing a query request to obtain a calculation operation type and a data distribution key; determining an input table and, when the input table contains a data distribution key, generating a local execution identifier; when the local execution identifier is detected, calculating the number of execution groups N based on the number of system CPU cores, the minimum number of scan rows, and the number of physical groups; performing group calculations to generate N to-be-executed groups; obtaining the amount of data for each to-be-executed group and generating an execution priority sequence; calculating the to-be-executed group with the highest priority in the execution priority sequence to generate an execution result; and generating a query result based on the corresponding execution results after all N to-be-executed groups have been executed. Implementation of this application can optimize resource utilization efficiency of aggregation and join operations in a distributed database system and ensure query performance.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a distributed database query method, system, medium and program product. Background Art

[0002] With the widespread adoption of big data applications, distributed database systems are required to handle increasingly large-scale data query and analysis tasks. Aggregation operations and table joins are the most common and resource-intensive of these tasks. To ensure query performance and resource efficiency, distributed database systems require the proper scheduling and allocation of computing resources.

[0003] In related technologies, distributed database systems typically use a data redistribution (shuffle) approach when performing aggregation and join operations, sending data with the same key value to the same compute node for processing. When the data distribution meets the computational requirements, the system chooses to perform the computation directly locally, avoiding the network overhead associated with data redistribution. During the execution phase, the system simultaneously activates all compute nodes to process data in parallel, with each node maintaining independent memory space for data caching and computation.

[0004] However, in high-concurrency query scenarios, starting all computing nodes simultaneously will significantly increase the peak system resource usage. Multiple execution threads processing different data sets in parallel will cause frequent CPU cache switching, reducing the cache hit rate and affecting query performance. Summary of the Invention

[0005] The present application provides a distributed database query method, system, medium and program product for optimizing resource utilization efficiency of aggregation and join operations in a distributed database system and ensuring query performance.

[0006] In the first aspect, the present application provides a query method for a distributed database, which is applied to a data processing system, and the method includes: receiving a query request and parsing it to obtain a calculation operation type and a corresponding data distribution key; determining the input table corresponding to the query request, and generating a local execution identifier when the input table contains a data distribution key; when the local execution identifier is detected, calculating the number of execution groups N based on the number of system CPU cores, the minimum number of scan rows and the number of physical groups; N is a positive integer; performing group calculation based on the distribution hash value of the data in the input table to generate N groups to be executed; obtaining the data volume of each group to be executed, and generating an execution priority sequence sorted in descending order according to the data volume; calculating the group to be executed with the highest priority in the execution priority sequence to generate an execution result; after all N groups to be executed are executed, generating a query result based on the corresponding execution result.

[0007] In the above embodiment, the data processing system dynamically calculates the appropriate number of execution groups based on CPU resources and data characteristics, and divides the data into multiple groups to be executed according to the distribution hash value. This can effectively control the peak memory usage and avoid competition for the CPU cache among multiple execution threads. At the same time, it ensures that execution groups with large amounts of data can be processed first, thereby improving the overall query performance and resource utilization efficiency.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of calculating the number of execution groups N based on the number of system CPU cores, the minimum number of scan lines and the number of physical groups when a local execution identifier is detected specifically includes: when a local execution identifier is detected, obtaining the number of system CPU cores and determining the corresponding single-thread grouping expected value; the single-thread grouping expected value is half of the number of system CPU cores; determining a first candidate value based on the product of the single-thread grouping expected value and a preset grouping coefficient; determining a second candidate value based on the ratio of the average number of rows of the scan node to the minimum number of scan lines; determining the smaller value of the first candidate value and the second candidate value as the maximum number of groups; determining the larger value of the maximum number of groups and the single-thread grouping expected value as the expected number of groups; determining the smaller value of the expected number of groups and the number of physical groups as the number of execution groups N.

[0009] In the above embodiment, the data processing system calculates the expected value of the single-thread group based on the number of CPU cores, thereby avoiding the generation of too small execution groups and preventing performance loss caused by excessive serialization.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the calculation operation type includes aggregation operations and join operations; determining the input table corresponding to the query request, and generating a local execution identifier when the input table contains a data distribution key, specifically includes: determining the input table corresponding to the query request; when the calculation operation type is a join operation, determining whether the input tables belong to the same collaborative distribution group; if so, generating a local execution identifier when it is determined that the input tables are the same table and the data distribution key contains a join condition column.

[0011] In the above embodiment, when processing a join operation, the data processing system first determines whether the input tables belong to the same co-distribution group and checks whether the data distribution key contains a join condition column, thereby reducing network transmission overhead while ensuring the accuracy of the query results.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after receiving a query request and parsing it to obtain the computing operation type and the corresponding data distribution key, the method also includes: determining the input table corresponding to the query request, and when the input table does not contain the data distribution key, obtaining the resource usage status of all computing nodes; calculating the available resource score of each computing node based on the resource usage status; sorting the computing nodes based on the available resource scores, and selecting the M computing nodes with the highest scores as candidate execution nodes; M is a positive integer; and allocating the data records of the input table to the M candidate execution nodes.

[0013] In the above embodiment, when the input table does not meet the local execution condition, the data processing system will analyze the resource usage status of all computing nodes to effectively balance the cluster load and improve the overall resource utilization efficiency.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after all N groups to be executed are executed, after the step of generating query results based on the corresponding execution results, the method also includes: collecting execution indicator information of each group to be executed; the execution indicator information includes actual execution time, resource usage and data processing rate; based on the execution indicator information, calculating the performance score of this query.

[0015] In the above embodiment, the data processing system collects detailed execution indicators of each execution group to evaluate the execution effect.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of calculating the performance score of the query based on the execution index information, the method also includes: when the performance score is lower than the preset performance threshold, calculating the resource utilization of each to-be-executed group based on the resource usage in the execution index information; adjusting the preset grouping coefficient based on the resource utilization, and adjusting the minimum number of scan rows based on the data processing rate in the execution index information.

[0017] In the above embodiment, when the performance score does not meet expectations, the system will analyze resource utilization and improve the efficiency of query execution.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the steps of obtaining the data volume of each to-be-executed group and generating an execution priority sequence sorted in descending order according to the data volume, the method further includes: obtaining execution status information of the to-be-executed group; when it is determined based on the execution status information that the processing progress of the current execution group is stagnant, splitting the unprocessed data of the current execution group into multiple sub-execution groups; inserting the multiple sub-execution groups into the execution priority sequence, and regenerating the execution priority sequence.

[0019] In the above embodiment, the data processing system monitors the status of each execution group in real time during the execution process. When it is found that the processing progress of a certain execution group is stagnant, its unprocessed data will be dynamically split into multiple smaller sub-execution groups to maintain the overall execution efficiency and ensure that the query can be completed stably.

[0020] In a second aspect, an embodiment of the present application provides a data processing system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the data processing system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a data processing system, enables the data processing system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a data processing system, causes the data processing system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] It is understandable that the data processing system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referenced to the beneficial effects of the corresponding methods and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By adopting a query request parsing and local execution judgment mechanism, as well as a group execution strategy based on data distribution characteristics, the system can accurately identify query scenarios suitable for local execution and organize query execution by dynamically calculating the number of execution groups and prioritizing them. This effectively solves the resource competition and performance degradation problems caused by the simultaneous operation of all execution units in related technologies, thereby achieving precise control of memory usage and efficient use of CPU cache.

[0026] 2. Due to the adoption of a dynamic group quantity calculation mechanism based on multi-dimensional parameters, the system can automatically determine the most appropriate number of execution groups for each query, effectively solving the resource waste and performance fluctuation problems caused by unreasonable execution parallelism settings in related technologies, thereby achieving precise allocation of execution resources and improving query performance.

[0027] 3. Due to the adoption of real-time monitoring of execution status and dynamic task splitting mechanism, the system can promptly detect and handle abnormal situations during the execution process, effectively solving the problem of overall performance degradation caused by stagnation of execution unit processing in related technologies, and thus achieving continuity and stability of query execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of a scenario in an embodiment of the present application;

[0029] Figure 2 This is a scenario diagram of a query method for a distributed database in related technologies;

[0030] Figure 3 This is a schematic diagram of a scenario of a distributed database query method in an embodiment of the present application;

[0031] Figure 4 This is a flow chart of a distributed database query method in an embodiment of the present application;

[0032] Figure 5 This is another scenario diagram of the distributed database query method in an embodiment of the present application;

[0033] Figure 6 This is another flowchart of the distributed database query method in an embodiment of the present application;

[0034] Figure 7 It is a schematic diagram of the structure of a physical device of the data processing system in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations of one or more of the listed items.

[0036] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0037] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0038] See also Figure 1 , Figure 1 This is a schematic diagram of a scenario in an embodiment of the present application; Figure 1 The following diagram shows a scenario in which a distributed database performs aggregation operations in related technologies. In this execution mode, the system contains multiple parallel execution units (represented by dotted boxes in the figure), and each execution unit is equipped with a scanning node and an aggregation node. The scanning node is responsible for reading the original data from the storage layer, while the aggregation node performs grouped statistical calculations on the data. Since the initial distribution of the data may not meet the needs of the aggregation calculation, the system needs to redistribute the data among the various execution units (the arrows marked as "redistribution" in the figure). Specifically, data records with the same aggregation key value need to be transferred to the same aggregation node for processing, which results in a large amount of network data transmission. After all aggregation nodes complete the calculation, the results are passed to the top-level output node to complete the entire query processing process.

[0039] In the related art, queries can be performed in a parallel processing manner. The following describes a scenario using a query method of a distributed database in the related art.

[0040] See also Figure 2 , Figure 2 This is a scenario diagram of a query method for a distributed database in related technologies; Figure 2 The figure shows a local execution scenario when the data distribution meets the computing requirements. In this case, since the data distribution key of the table contains the columns required for aggregation calculations, the data read by each scanning node can be processed directly in the local aggregation node without the need for data redistribution. Although this execution method avoids network transmission overhead, it still has significant problems: all execution units will start and run at the same time, and each aggregation node needs to allocate and maintain independent memory space to store intermediate calculation results. In large-scale data processing scenarios, this parallel execution method will cause the system's memory usage to rise sharply. At the same time, multiple execution threads processing different data sets in parallel will cause the CPU cache to switch frequently, reduce the cache hit rate, and affect computing performance.

[0041] The distributed database query method in the embodiment of the present application is used to improve the overall query efficiency by giving priority to processing large data volume groups. The following describes a scenario in which the distributed database query method in the present application is used.

[0042] In response to the above problems, this application proposes an optimized execution strategy, such as Figure 3 See Figure 3 , Figure 3 It is a scenario diagram of the query method of the distributed database in the embodiment of the present application. The system organizes multiple execution units into a waiting queue, and each execution unit in the queue (to-be-executed group 1 to to-be-executed group N in the figure) contains a complete combination of scanning nodes and aggregation nodes. The system determines the priority of the execution group according to the amount of data, and gives priority to scheduling the execution group with a larger amount of data. During the specific execution process, the system only activates one execution group at a time for calculation. When the execution group completes the processing, it will schedule the next execution group in the queue to start work. This serialized execution method ensures that only one aggregation node is performing calculations at the same time, significantly reducing the peak memory usage of the system. At the same time, since the competition for the CPU cache among multiple execution threads is avoided, the efficiency of cache utilization is improved.

[0043] It can be seen that the query method of the distributed database in the embodiment of the present application can effectively solve the problems of excessive resource peaks and frequent cache switching caused by parallel execution in related technologies, thereby achieving improved query performance.

[0044] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 4 , which is a flow chart of a distributed database query method in an embodiment of the present application.

[0045] S401: Receive a query request and parse it to obtain a calculation operation type and a corresponding data distribution key.

[0046] Among them, the query request represents the data query instruction sent by the user or application to the data processing system, which includes SQL statements or other query language forms; the calculation operation type refers to the specific calculation operation involved in the query request, such as aggregation operations (SUM, COUNT, AVG, etc.) or connection operations (JOIN); the data distribution key represents the column or column combination specified when the table is created for distributing data to different computing nodes, which is specified through the "DISTRIBUTED BY HASH" statement; the parsing process refers to the process of converting the query request into an internal representation that can be recognized and processed by the data processing system.

[0047] When a data processing system receives a database query request from a user, it must first parse the query. Specifically, the system performs lexical and syntactic analysis on the input SQL statement to identify the query components. It then extracts the type of calculation operation within the query. For example, for a query like "SELECT k1, k2, sum (v FROM t1 GROUP BY k1, k2)," the system identifies it as a GROUP BY operation with a SUM aggregation. The system also parses the table structures involved in the query, obtaining the data distribution key information specified when these tables were created. This parsing process generates an internal data structure that contains the complete query semantics, providing the foundation for subsequent query optimization and execution.

[0048] During actual execution, query request parsing may encounter syntax errors, unknown identifiers, data type mismatches, and other issues. This is especially true when processing queries containing complex expressions, which can lead to parsing ambiguities. To address this, the data processing system employs a priority-based parsing strategy: First, an operator precedence table is established to clearly define the associativity and precedence of various operators. Then, during the parsing process, when encountering potentially ambiguous expressions, the system determines the correct parsing order based on priority rules. For example, when parsing an expression like "SELECT a + b * c," the system prioritizes multiplication operations over addition operations. The system also maintains a symbol table that stores recognized identifiers and their attributes, enabling rapid detection of unknown identifiers and type mismatches. For detected errors, the system generates detailed error information, including the error location, error type, and possible correction suggestions, to help users quickly locate and resolve the issue.

[0049] S402: Determine the input table corresponding to the query request, and generate a local execution identifier when the input table contains a data distribution key.

[0050] The input table represents the data table involved in the query, including the main table and associated tables; containing data distribution keys means that the distribution key columns specified in the table creation statement of the input table include the columns required for the query operation; the local execution flag represents a Boolean flag used to indicate whether the query can be executed directly on the local node without data redistribution; the determination process refers to the process of determining whether the query conditions match the data distribution by analyzing the metadata information of the table.

[0051] After the data processing system completes the parsing of the query request, it needs to determine whether local computing optimization can be performed. Specifically, the data processing system first obtains all input table information involved in the query, including metadata such as the physical storage structure of the table and the distribution key setting. The system then handles two main scenarios separately: for aggregation operations, it checks whether the columns in the GROUP BY clause are completely included in the distribution key of the table; for join operations, it is necessary to verify whether all tables involved in the join belong to the same co-distribution group and whether the columns in the join conditions are included in the distribution key of the table. Only when these conditions are met will the system generate a local execution flag, indicating that the query can avoid data redistribution. This judgment process takes into account multi-table associations, subqueries, and other situations in complex queries to ensure that the generated execution plan is correct and efficient.

[0052] In some embodiments, local execution conditions can be determined through various approaches. Alternatively, the data processing system can employ a rule-based approach, first constructing a set containing query condition columns and distribution key columns, and then determining whether the local execution conditions are met based on the inclusion relationship within the set. The specific process includes: extracting groupby columns or join condition columns from the query to construct the condition set; extracting distribution key columns from the table metadata to construct the distribution key set; and determining whether the condition set is a subset of the distribution key set. Alternatively, the data processing system can employ a graph-based analysis approach, representing the distribution relationships between tables as a graph structure and analyzing data distribution compatibility using a graph traversal algorithm. This approach is particularly suitable for processing complex multi-table join queries. The specific steps include: constructing a graph structure representing the relationships between tables; analyzing the connected components in the graph to determine data distribution compatibility; and determining whether local execution is possible based on the analysis results. It is understood that other analysis approaches can also be employed to determine local execution conditions, which are not limited here. Furthermore, the system must also consider various complexities within the query, such as expression evaluation and subqueries, which require more complex semantic analysis and judgment logic.

[0053] During actual execution, local execution conditions may encounter difficulties determining distribution key matching due to complex expressions. For example, when query conditions include function calls or complex expressions, simple column name matching may not accurately determine whether local execution conditions are met. To address this issue, the data processing system employs an expression normalization strategy: first, expressions in query conditions are converted to a standard form, which includes operations such as function call expansion, constant folding, and expression simplification. The normalized expressions are then analyzed to determine whether they can exploit the data's distribution characteristics. The system also maintains a function attribute registry to record the impact of different functions on data distribution characteristics. For example, some functions may disrupt data distribution characteristics, making local execution impossible. Through this fine-grained analysis, the system can more accurately determine whether a query meets local execution conditions.

[0054] S403: When a local execution flag is detected, the number of execution groups N is calculated according to the number of system CPU cores, the minimum number of scan lines, and the number of physical groups.

[0055] The number of system CPU cores indicates the number of processor cores available for computing in the data processing system. The minimum number of scan rows refers to the minimum data volume threshold processed by a single execution group, which is used to avoid generating execution groups that are too small. The number of physical groups indicates the number of data distribution buckets (BUCKETS) specified when creating the table. The number of execution groups N indicates the final number of groups to be executed. The calculation process refers to the process of obtaining the optimal number of execution groups based on a comprehensive evaluation of multiple parameters.

[0056] After confirming that local execution is possible, the data processing system needs to rationally plan the allocation of execution resources. Specifically, the data processing system first obtains the number of currently available CPU cores, and usually sets the single-thread parallel execution degree to half the number of CPU cores to reserve resources for processing other tasks. The system then evaluates the actual data processing situation, including the average number of rows scanned by the node and the preset minimum scan row threshold, to avoid generating too small execution groups that result in additional scheduling overhead. At the same time, the system also needs to consider the limitations of the physical storage level to ensure that the number of execution groups does not exceed the number of buckets specified when the table was created to maintain the integrity of the data distribution. In this process, the system will weigh the degree of parallelism and resource consumption, dynamically adjust the number of groups, and ultimately obtain an execution group number N that balances parallel processing capabilities and system overhead.

[0057] In some embodiments, calculation of the number of execution groups can be achieved through various methods: Optionally, the data processing system can employ a resource assessment-based calculation method, including the following steps: calculating the expected single-thread parallelism (expectedGroup = number of CPU cores / 2); calculating the initial grouping upper limit (maxGroup = min(expectedGroup * groupScale, configMaxGroup)) based on a preset grouping coefficient; adjusting maxGroup = min(avgScanRows / minScanRows, maxGroup) based on the actual data volume; and considering physical grouping limitations to obtain the final number of execution groups (N = min(maxGroup, physicalBuckets)). Optionally, the data processing system can also employ an adaptive calculation method, dynamically adjusting grouping parameters based on historical execution data, including the following steps: collecting historical query execution statistics; analyzing execution efficiency under different numbers of groups; and dynamically adjusting the grouping coefficient and minimum scan row threshold based on the analysis results. It is understood that other calculation methods can also be employed to determine the number of execution groups, and are not limited here.

[0058] During actual execution, calculating the number of execution groups may encounter load imbalance caused by data skew. For example, when data is unevenly distributed across physical buckets, simply dividing execution groups by the number of buckets may result in some execution groups processing significantly more data than others. To address this situation, the data processing system employs a data-aware grouping strategy: First, the data volume in the physical buckets is sampled and counted to assess the evenness of the data distribution. Based on the statistical results, the original physical groups are reorganized, merging physical buckets with similar data volumes into a single execution group or splitting physical buckets with excessively large data volumes into multiple execution groups. The system also sets a data volume difference threshold; when the data volume difference between execution groups exceeds the threshold, regrouping is triggered. This approach effectively balances the processing load across execution groups, improving overall execution efficiency. Simultaneously, the system continuously monitors execution status and dynamically adjusts the grouping strategy when necessary.

[0059] S404: Perform group calculation based on the distributed hash value of the data in the input table to generate N groups to be executed.

[0060] Among them, the distribution hash value represents the hash value calculated on the distribution key column of the data record, which is used to determine the physical storage location of the data; group calculation refers to the process of dividing data into different execution groups according to the hash value; the to-be-executed group represents an execution unit containing a set of related data records, with independent scanning nodes and computing nodes.

[0061] After determining the number of execution groups, the data processing system needs to perform specific grouping based on the distribution characteristics of the data. Specifically, the data processing system first uses a consistent hashing algorithm to calculate the hash value of the distribution key column value to ensure that the same distribution key value can obtain the same hash value. Then, based on the number of execution groups N calculated in the previous step, the hash value space is evenly divided into N ranges, and each range corresponds to a group to be executed. For each data record in the input table, the system determines the group to be executed to which it belongs based on the hash value of its distribution key. This process needs to ensure the integrity and consistency of the data, that is, data records with the same distribution key value must be assigned to the same group to be executed to ensure the correctness of the calculation results. At the same time, the system also needs to create independent scanning nodes and computing nodes for each group to be executed to prepare for subsequent parallel processing.

[0062] S405 : Obtain the data volume of each to-be-executed group, and generate an execution priority sequence sorted in descending order according to the data volume.

[0063] Among them, the data volume refers to the total number of data records contained in each to-be-executed group; the execution priority sequence refers to the queue of to-be-executed groups sorted according to the size of the data volume, which is used to determine the execution order; the order from large to small indicates that the to-be-executed group with a large amount of data has a higher execution priority.

[0064] After completing data grouping, the data processing system needs to rationally arrange the processing order for each execution group. Specifically, the data processing system first traverses all pending execution groups and counts the actual number of data records contained in each execution group. This statistical process takes into account the size and complexity of the data records, and adjusts the weights for records containing large fields or complex data types. The system then sorts all pending execution groups in descending order based on their data volume to generate a priority queue. This sorting strategy ensures that execution groups with larger data volumes receive priority processing resources, helping to improve overall execution efficiency. At the same time, the system also records other characteristic information for each execution group, such as data distribution characteristics and estimated execution time, to provide a basis for subsequent execution scheduling.

[0065] In some embodiments, the determination of execution priority can be achieved in a variety of ways: Optionally, the data processing system can adopt a multi-dimensional scoring method, the steps of which include: calculating the basic data volume score; adjusting it by considering the data complexity factor; weighting it in combination with historical execution statistics; generating a comprehensive priority score; and sorting it according to the score. Optionally, the data processing system can also adopt an adaptive priority strategy to dynamically adjust the priority by monitoring the system resource status in real time. The specific steps include: monitoring the system resource usage; evaluating the resource requirements of the execution group; and adjusting the priority according to the resource matching degree. It is understandable that other methods can also be used to achieve the determination of execution priority, which are not limited here.

[0066] During actual execution, priority sorting may encounter resource estimation deviations caused by complex queries. For example, when a query contains complex calculation expressions or user-defined functions, the data volume alone may not accurately reflect the actual processing complexity. To address this situation, the data processing system adopts a priority calculation strategy assisted by a cost model: first, a basic cost model for query operations is established, including dimensions such as CPU calculation cost, memory usage cost, and I / O cost; then, a cost estimate is performed for each operation in the query, and the comprehensive cost of the execution group is calculated based on the data volume; finally, the execution priority is determined based on the comprehensive cost. The system also maintains cost statistics for historical executions to continuously optimize the accuracy of cost estimates. When a significant deviation is found between the actual execution cost and the estimated value, the system triggers dynamic adjustment of the cost model.

[0067] S406: Calculate the to-be-executed group with the highest priority in the execution priority sequence to generate an execution result.

[0068] Among them, executing calculations means performing specific processing operations on the data in the pending execution group, such as aggregation calculations or connection operations; the highest priority refers to the pending execution group that is at the front of the execution priority sequence; and the execution result refers to the intermediate result data generated after a single pending execution group completes the calculation.

[0069] After determining the execution priority, the data processing system begins to process each to-be-executed group sequentially. Specifically, the data processing system first obtains the to-be-executed group with the highest priority from the execution priority sequence, activates the scanning node of the execution group, and starts data reading. The scanning node will adopt a batch reading strategy to load the data into the memory. The system then performs the corresponding calculation operation based on the query type. For example, for aggregation operations, a hash table will be built to store the grouped data and perform aggregation calculations; for join operations, a join index will be established and join matching will be performed. During the calculation process, the system will monitor resource usage in real time, including memory usage, CPU utilization, etc., to ensure that the processing of a single execution group does not put too much pressure on the system. After the calculation is completed, the system will store the execution results in the specified result buffer and release the relevant computing resources.

[0070] In some embodiments, computational processing of the pending group can be implemented in a variety of ways: Optionally, the data processing system can use a pipeline processing method, including the following steps: starting a scanning pipeline to read data in batches; activating a computational pipeline to process the data; maintaining computational status to ensure data consistency; generating interim results; and merging final execution results. Optionally, the data processing system can also use a vectorized execution method to improve computational efficiency through batch data processing, including the following steps: organizing data into a vector format; applying vectorized computational operations; and optimizing memory access patterns. It is understood that other execution methods can also be used to implement computational processing of the pending group, which is not limited here.

[0071] During actual execution, computational processing may encounter the risk of memory overflow. In particular, when processing aggregation operations involving a large number of groups or large table joins, intermediate results may occupy excessive memory. To address this situation, the data processing system employs an adaptive memory management strategy: first, the execution group's memory requirements are evaluated and a reasonable upper limit for memory usage is set; then, memory usage is monitored in real time during execution, triggering an overflow handling mechanism when a preset threshold is reached. Specific handling methods include: temporarily storing some intermediate results on disk and retaining hot data in memory; using partitioning to split large computational tasks into multiple small batches for serial processing; and dynamically adjusting computational parallelism to avoid processing too much data simultaneously. The system also maintains statistical information on memory usage to optimize resource allocation strategies for subsequent execution groups.

[0072] S407: After all N to-be-executed groups have been executed, a query result is generated based on the corresponding execution results.

[0073] Among them, execution completion means that all N groups to be executed have completed their respective calculation processing; query result means the final output data obtained by merging the execution results of all execution groups; generation process refers to the operation process of summarizing, merging and post-processing multiple execution results.

[0074] After the data processing system completes processing of all pending execution groups, it needs to consolidate the final results. Specifically, the data processing system first checks the completion status of all execution groups to ensure that there are no execution failures or timeouts. Then, the appropriate result merging strategy is selected based on the query type. For example, for aggregation operations, the local aggregation results of each execution group need to be merged and calculated; for join operations, the integrity and uniqueness of the results need to be ensured. During the merging process, the system will perform necessary data conversion and formatting to ensure that the final output query results meet the format requirements specified by the user. At the same time, the system will also collect statistical information during the execution process, including processing time, resource usage, etc., for subsequent query optimization.

[0075] During the actual execution process, the result merging may encounter data consistency issues. For example, when the results generated by different execution groups overlap or conflict, a simple merge may lead to inaccurate results. To address this situation, the data processing system adopts a transactional result merging strategy: first, a version identifier is established for each execution result, recording its generation time and dependencies; then, during the merge process, version consistency is checked to ensure that the merge operations are performed in the correct order. The system also implements a rollback mechanism. When an abnormality is found during the merge process, it can roll back to the previous consistent state and merge again. For scenarios where deduplication is required, the system maintains a set of globally unique identifiers to ensure that duplicate data does not appear in the final result. At the same time, the system also logs the merge process to facilitate problem location and result verification.

[0076] The following is a more detailed description of the process of the method provided by this implementation. Figure 6 , is another flow chart of the distributed database query method in an embodiment of the present application.

[0077] S601: Receive a query request and parse it to obtain a calculation operation type and a corresponding data distribution key.

[0078] Referring to step S401 , the data processing system parses the query request.

[0079] In some embodiments, the data processing system performs candidate planning, that is, the data processing system determines the input table corresponding to the query request, and obtains the resource usage status of all computing nodes when the input table does not contain the data distribution key; calculates the available resource score of each computing node based on the resource usage status; sorts the computing nodes based on the available resource score, and selects the M computing nodes with the highest scores as candidate execution nodes; and distributes the data records of the input table to the M candidate execution nodes.

[0080] Among them, the resource usage status represents the current resource consumption of the computing node, including indicators such as CPU utilization and memory occupancy; the available resource score represents the node availability score obtained by comprehensive calculation based on multiple resource dimensions; the candidate execution node represents the computing node selected to perform data processing tasks; the data record allocation represents the process of allocating the data of the input table to different execution nodes according to specific rules; M is a positive integer, which represents the number of candidate nodes determined by the system based on the data scale and resource conditions.

[0081] When the data processing system discovers that the input table does not contain the data distribution key, it needs to re-plan the data processing strategy. Specifically, the data processing system first obtains the real-time resource usage status of all computing nodes in the cluster through the resource monitoring component. These status data include indicators in multiple dimensions such as CPU utilization, memory occupancy, disk I / O load, and network bandwidth usage. The data processing system then calculates a comprehensive available resource score for each computing node based on these resource indicators. The score calculation takes into account the weights and threshold constraints of different resource dimensions. Next, the data processing system sorts all computing nodes in descending order according to the available resource score, and selects the M nodes with the highest scores as candidate execution nodes based on the data processing requirements. Finally, the data processing system uses the principle of data volume balance to distribute the data records of the input table to these M candidate execution nodes.

[0082] In some embodiments, node selection and data allocation can be achieved in a variety of ways: Optionally, the data processing system can adopt a multi-dimensional weighted resource scoring method, by standardizing various resource indicators, combining the resource importance weights to calculate the comprehensive score of the node, and setting a dynamic resource usage threshold to screen candidate nodes. Optionally, the data processing system can also adopt a load prediction allocation method, by analyzing the historical load change trend and data processing characteristics of the node, predicting the resource demand during the execution period, so as to achieve more reasonable data allocation. It is understandable that other methods can also be used to achieve node selection and data allocation. In addition, the system also needs to consider the processing mechanism of abnormal situations such as node failures and network delays.

[0083] During actual execution, candidate node selection may encounter a mismatch between resource evaluation and data characteristics. For example, when the resource demand pattern during data processing differs significantly from the initial evaluation, node selection based on static resource status may not achieve optimal results. To address this situation, the data processing system adopts an adaptive resource evaluation strategy: first, a prediction model that includes data characteristics and resource consumption patterns is constructed. This model uses machine learning methods to learn resource usage patterns from historical execution data; then, based on the data characteristics of the current query, its resource consumption trends on different nodes are predicted; finally, the node resource score calculation method is dynamically adjusted based on the prediction results. The system also implements a load balancing mechanism during execution. When a significant deviation in node load is found, it can trigger data redistribution to ensure balanced processing load.

[0084] S602: Determine the input table corresponding to the query request, and generate a local execution identifier when the input table contains a data distribution key.

[0085] Referring to step S402 , the data processing system generates a local execution identifier.

[0086] In some embodiments, the data processing system will perform similar judgments, that is, the data processing system will determine the input table corresponding to the query request; when the calculation operation type is a join operation, it will determine whether the input tables belong to the same collaborative distribution group; if so, if it is determined that the input tables are the same table and the data distribution key contains a join condition column, a local execution identifier will be generated.

[0087] Among them, the input table corresponding to the query request represents the set of data tables participating in the query operation; the join operation represents the database operation of performing association calculations on multiple tables; the collaborative distribution group refers to a set of tables with the same distribution characteristics; the data distribution key represents the column used for data distribution specified when the table is created; the join condition column represents the column used to associate tables in the join operation; the local execution flag represents a flag that allows execution on the local node without data redistribution.

[0088] After parsing the query request, the data processing system needs to determine whether local execution optimization can be performed. Specifically, the data processing system first identifies all input tables involved in the query and obtains metadata information for these tables. When it is found that the query contains a join operation, the data processing system will further check whether these input tables belong to the same co-distribution group, that is, whether they use the same distribution strategy and distribution key. For self-join scenarios (that is, the join operation involves the same table), the data processing system will also verify whether the table's distribution key contains the columns used in the join condition. Only when all these conditions are met will the data processing system generate a local execution flag, indicating that the join operation can be completed locally without data redistribution.

[0089] In some embodiments, local execution judgment of the connection operation can be implemented in a variety of ways: Optionally, the data processing system can adopt a static analysis method based on metadata, and the specific steps include: obtaining the distribution key definition of the table from the system catalog; parsing the connection condition to extract the columns involved; establishing a mapping relationship between the distribution key and the connection condition column; verifying the integrity of the mapping relationship; and checking the compatibility of the distribution strategy. Optionally, the data processing system can also adopt a dynamic analysis method based on the execution plan to verify the feasibility of local execution by simulating data distribution, and the specific steps include: generating sample data distribution; analyzing the data flow path; evaluating the data movement cost; and determining the optimal execution strategy. It is understandable that other methods can also be used to implement local execution judgment of the connection operation, which are not limited here. In addition, the system also needs to consider the judgment logic in complex connection scenarios, such as multi-table connections, function calls included in conditional expressions, etc.

[0090] For the connection operation, this application also adopts the optimization strategy of group execution. Figure 5 , Figure 5is another scenario diagram of the query method of the distributed database in the embodiment of the present application; Figure 5 As shown in the figure, under the traditional execution method (the upper half of the figure), the system simultaneously launches multiple execution units for join operations, each of which consists of two scan nodes and one join node. This parallel execution method also brings issues with memory usage and CPU cache efficiency. In the optimized solution (the lower half of the figure), the system organizes these execution units into waiting queues and schedules each execution unit in sequence according to a predetermined priority order. It should be noted that this optimization method applies not only to self-join operations on the same table, but also to join operations between different tables belonging to the same co-distribution group, as long as the tables have the same data distribution characteristics.

[0091] This grouped execution approach allows the system to effectively control peak resource usage and improve computing efficiency. Specifically, when processing a query involving aggregation or join operations, the system first parses the query request to determine whether it meets the conditions for local execution. If so, it calculates the appropriate number of execution groups based on factors such as the system's number of CPU cores and data distribution characteristics. These execution groups are then prioritized based on data size, ultimately completing query processing through sequential scheduling.

[0092] S603: When the local execution flag is detected, the number of CPU cores in the system is obtained, and the corresponding single-thread group expected value is determined.

[0093] The number of system CPU cores indicates the number of processor cores currently available in the data processing system; the expected value of a single-threaded group indicates the ideal execution parallelism calculated based on the number of CPU cores; and the detection of a local execution flag indicates confirmation that the current query meets the local execution conditions.

[0094] After confirming that the query can be executed locally, the data processing system first needs to calculate the appropriate degree of parallelism. Specifically, the data processing system will first obtain the number of available CPU cores. Considering system stability and the resource requirements of other tasks, the expected value of single-threaded grouping is typically set to half the number of CPU cores. This setting ensures the parallel efficiency of query processing while reserving sufficient computing resources for other tasks in the system. The system also considers the actual CPU load status and dynamically adjusts this ratio when necessary to ensure the rational allocation of system resources.

[0095] In some embodiments, calculation of the expected value of a single-threaded group can be achieved through various methods: Optionally, the data processing system can employ a dynamic evaluation method, including the following steps: obtaining system CPU usage statistics; analyzing resource consumption patterns of historical queries; adjusting allocation ratios based on the current system load; calculating a final expected value; and applying minimum and maximum limits. Optionally, the data processing system can also employ an adaptive calculation method, dynamically adjusting the expected value by monitoring system performance indicators in real time. It is understood that other methods can also be employed to achieve calculation of the expected value of a single-threaded group, and these are not limited here.

[0096] In actual execution, the calculation of expected values ​​for single-threaded groups may encounter fluctuations in system load. For example, when multiple resource-intensive tasks are running simultaneously in the system, a simple fixed-ratio calculation may not accurately reflect the available resources. To address this situation, the data processing system adopts a resource-aware computing strategy: first, a system resource usage monitoring mechanism is established to collect real-time metrics such as CPU utilization and memory usage. Then, a resource scoring model is constructed based on these metrics to dynamically adjust the calculation ratio of the expected value for single-threaded groups. The system also maintains historical resource usage statistics to predict short-term resource change trends and optimize the calculation of expected values. When a significant change in system load is detected, the system triggers a recalculation of the expected value to ensure that the execution parallelism remains within a reasonable range.

[0097] S604: Determine a first candidate value according to the product of the single-thread grouping expected value and a preset grouping coefficient.

[0098] Among them, the preset grouping coefficient represents a system configuration parameter used to adjust the number of groups, which is usually greater than 1; the first candidate value represents a preliminary grouping upper limit calculated by the single-thread expected value and the grouping coefficient; the product calculation represents the operation of multiplying the single-thread grouping expected value by the preset grouping coefficient.

[0099] After obtaining the expected single-thread grouping value, the data processing system needs to scale it up based on the system's actual load capacity. Specifically, the data processing system first retrieves a preset grouping coefficient from the configuration. This coefficient is typically determined based on the system's historical execution data and performance test results. The expected single-thread grouping value is then multiplied by this coefficient to obtain the first candidate value. This calculation process takes into account resource fluctuations and task scheduling overhead during query execution, providing a relatively loose upper limit for subsequent grouping calculations.

[0100] In actual execution, the use of preset grouping coefficients may encounter issues with varying query complexity. To address this, the data processing system employs a query-aware coefficient adjustment strategy: It first analyzes query complexity characteristics, including the number of tables involved and the complexity of the computational expressions. It then dynamically adjusts the grouping coefficient based on these characteristics, using a smaller coefficient for complex queries and a larger coefficient for simpler queries.

[0101] S605 : Determine a second candidate value according to a ratio of an average number of rows of scanned nodes to a minimum number of scanned rows.

[0102] Among them, the average number of rows in the scan nodes represents the average number of data rows in all scan nodes, which is used to reflect the overall data distribution; the minimum number of scan rows represents the minimum number of data records that a single execution group needs to process as set by the system, which is used to prevent the execution group from being too small; the second candidate value represents the upper limit of the number of groups calculated by the data volume dimension; the ratio represents the quotient obtained by dividing the average number of rows in the scan nodes by the minimum number of scan rows.

[0103] When determining the number of groups, the data processing system needs to make reasonable plans based on the actual data volume characteristics. Specifically, the data processing system first traverses all scan nodes, counts the number of data records actually contained in each node, and calculates the average. The system then obtains a pre-configured minimum scan row threshold, which is usually set based on the system's processing capabilities and historical execution experience. The system then divides the average number of rows in the scan node by the minimum number of scan rows to obtain the second candidate value. This calculation process ensures that each execution group has access to a sufficient amount of data for processing, avoiding the problem of excessive task scheduling overhead or reduced execution efficiency due to overly detailed grouping.

[0104] In some embodiments, group calculation based on data volume can be implemented in a variety of ways: Optionally, the data processing system can adopt an adaptive calculation method, and the specific steps include: collecting data statistical information of all scanning nodes; analyzing the uniformity of data distribution; dynamically adjusting the minimum number of scan rows according to data distribution characteristics; calculating a preliminary grouping ratio; applying upper and lower limit constraints to obtain the final second candidate value. Optionally, the data processing system can also adopt a multi-dimensional evaluation method, in addition to considering the number of rows, it also combines the complexity and processing cost of the data for comprehensive calculation, and the specific steps include: evaluating the average size of data records; analyzing the complexity of the data type; calculating the processing cost weight; and calculating the grouping ratio based on multiple factors. It is understandable that other methods can also be used to implement group calculation based on data volume, which is not limited here.

[0105] S606: Determine the smaller value between the first candidate value and the second candidate value as the maximum number of groups.

[0106] Among them, the first candidate value represents the grouping upper limit of the system resource dimension calculated by the single-thread grouping expectation value and the preset grouping coefficient; the second candidate value represents the grouping upper limit of the data volume dimension calculated by the average number of rows of the scanning node and the minimum number of scanned rows; the maximum number of groups represents the upper limit of the number of groups determined after comprehensively considering the system resource limitations and data processing requirements; the smaller value means selecting the smaller one among multiple candidate values ​​as the final result.

[0107] After obtaining candidate values ​​for the two dimensions, the data processing system needs to perform unified restriction processing. Specifically, the data processing system first compares the first candidate value with the second candidate value and selects the smaller value as the maximum number of groups. This selection strategy adopts a conservative principle, satisfying both system resource constraints and ensuring data processing efficiency. If the larger of the first and second candidate values ​​is used, it may lead to overallocation of system resources or insufficient data in a single execution group. By selecting the smaller value, the system can avoid wasting resources while ensuring execution efficiency.

[0108] In some embodiments, the determination of the maximum number of groups can be achieved in a variety of ways: Optionally, the data processing system can adopt a progressive comparison method, and the specific steps include: setting the initial maximum number of groups to the maximum value supported by the system; comparing it with the first candidate value and taking the smaller value; comparing the result of the previous step with the second candidate value and taking the smaller value; checking whether the result meets the minimum grouping requirement of the system; rounding up if necessary. Optionally, the data processing system can also adopt a multi-threshold constraint method, in addition to considering two candidate values, also introduce other system limiting factors, and the specific steps include: collecting the current resource limitation parameters of the system; calculating the grouping upper limit under each limitation; uniformly comparing all upper limit values; and selecting the final maximum number of groups. It is understandable that other methods can also be used to achieve the determination of the maximum number of groups, which are not limited here.

[0109] S607: Determine the larger value of the maximum number of groups and the expected value of single-thread grouping as the expected number of groups.

[0110] Among them, the maximum number of groups represents the upper limit of grouping obtained by comparing the first candidate value and the second candidate value; the expected value of single-threaded grouping represents the ideal execution parallelism calculated based on the number of CPU cores; the expected number of groups represents the target number of groups obtained on the basis of ensuring the minimum parallelism requirement; the larger value means selecting the larger one of the two input values ​​as the final result, which is used to ensure that the execution parallelism is not lower than the benchmark value set by the system.

[0111] After determining the maximum number of groups, the data processing system must ensure that the resulting execution parallelism is not too low. Specifically, the data processing system first compares the maximum number of groups with the expected number of groups per thread and selects the larger value as the expected number of groups. This selection strategy uses a guaranteed minimum number of groups to ensure that query execution meets the minimum parallel processing requirement, even when the data volume is small or system resources are limited. If the maximum number of groups is used directly, it may be too small to fully utilize the system's parallel processing capabilities, affecting query performance.

[0112] In some embodiments, the desired number of groups can be determined in a variety of ways: Optionally, the data processing system can adopt a performance evaluation-based approach, specifically including: constructing a query performance prediction model; analyzing the expected execution time under different numbers of groups; evaluating the impact of parallelism on performance; comprehensively considering resource consumption and performance benefits; and determining the final desired number of groups. Optionally, the data processing system can also adopt an adaptive adjustment approach to dynamically adjust the desired number of groups by monitoring system performance indicators in real time, specifically including: monitoring system resource utilization; analyzing query execution efficiency; adjusting parallelism based on performance feedback; and updating the desired number of groups. It is understood that other approaches can also be used to determine the desired number of groups, which are not limited here.

[0113] During actual execution, determining the desired number of groups may encounter the problem of diminishing returns. For example, once the number of groups reaches a critical value, further increasing the degree of parallelism may not lead to significant performance improvements, but may instead increase system overhead. To address this issue, the data processing system employs a parallel benefit evaluation strategy: first, a benefit model for parallel processing is established, incorporating factors such as increased processing speed and increased resource consumption. Then, historical execution data is used to analyze the input-output ratio under different degrees of parallelism. Finally, the desired number of groups is dynamically adjusted based on the benefit evaluation results. The system also maintains a parallelism optimization log to record the optimal parallelism configuration for different query types and data scales. When a new query request is detected, the system quickly determines the appropriate desired number of groups by referring to historical experience with similar queries. Simultaneously, the system continuously monitors execution performance and adjusts the parallelism configuration accordingly.

[0114] S608: Determine the smaller value between the expected number of packets and the physical number of packets as the execution number N of packets.

[0115] The expected number of groups indicates the target number of groups obtained after ensuring the minimum parallelism requirement; the physical number of groups indicates the number of buckets specified when the table is created; the number of executed groups N indicates the actual number of executed groups ultimately determined; the smaller value indicates that the smaller value between the expected number of groups and the physical number of groups is selected as the final result, ensuring that the number of executed groups does not exceed the bucket limit of physical storage.

[0116] After determining the desired number of groups, the data processing system must also consider physical storage constraints. Specifically, the data processing system first obtains the number of buckets specified when the input table was created. This number represents the number of partitions in the physical storage layer. It then compares the desired number of groups with the number of physical groups, selecting the smaller value as the final number of execution groups, N. This selection strategy ensures that the execution plan matches the underlying storage structure, avoiding reduced data access efficiency or additional data sorting overhead caused by the number of groups exceeding the number of physical buckets.

[0117] In some embodiments, the final number of execution groups can be determined in a variety of ways: Optionally, the data processing system can use a bucket-aware calculation method, and the specific steps include: analyzing the data distribution of physical buckets; evaluating the degree of data skew between buckets; adjusting the grouping strategy according to the data distribution characteristics; calculating the actual combination of buckets that can be merged; and determining the final number of execution groups. Optionally, the data processing system can also use a dynamic tuning method to optimize the determination of the number of groups by analyzing historical execution effects, and the specific steps include: collecting execution statistics under different grouping configurations; analyzing the relationship between the number of groups and execution efficiency; establishing a grouping effect evaluation model; and dynamically adjusting the grouping strategy. It is understandable that other methods can also be used to determine the number of execution groups, which are not limited here.

[0118] During actual execution, determining the number of execution groups may encounter issues with uneven physical bucket capacity. For example, when data is unevenly distributed across physical buckets, simply limiting grouping by the number of buckets can result in an uneven processing load. To address this issue, the data processing system employs a bucket capacity-aware optimization strategy: First, a statistical analysis of the data volume distribution across physical buckets is performed to calculate the actual data volume and processing cost for each bucket. Next, based on data distribution characteristics, adjacent small-capacity buckets are merged or large-capacity buckets are appropriately split. Finally, the actual number of execution groups is determined based on the optimized bucket combination. The system also maintains a bucket status monitoring mechanism that continuously records data trends across buckets. Significant changes in data distribution trigger a reassessment and adjustment of the grouping strategy. This allows the system to achieve more balanced and efficient data processing within physical storage constraints.

[0119] S609: Perform group calculation based on the distributed hash value of the data in the input table to generate N groups to be executed.

[0120] Referring to step S404 , the data processing system generates a to-be-executed group.

[0121] S610: Obtain the data volume of each to-be-executed group, and generate an execution priority sequence sorted in descending order according to the data volume.

[0122] Referring to step S405 , the data processing system generates a priority sequence.

[0123] S611. Obtain execution status information of the to-be-executed group.

[0124] Among them, the execution status information represents the real-time status data of each pending execution group during the operation process; the pending execution group represents the data set divided according to the distribution hash value and waiting to be processed; the acquisition process represents the operation process of the system regularly collecting and updating the execution status data; the status information includes key indicators such as processing progress, resource usage, and execution time.

[0125] During execution, the data processing system needs to monitor the operating status of each pending execution group in real time. Specifically, the data processing system establishes a status monitoring mechanism for each pending execution group, regularly collecting execution metrics across multiple dimensions. These metrics include the number of processed data records, data processing rate, CPU utilization, memory usage, and I / O wait time. The system integrates this information into a comprehensive execution status view, which is used for subsequent execution anomaly detection and handling strategy adjustments. The system also records key events during execution, such as data loading completion, processing phase switching, and other important time points.

[0126] During actual execution, execution status monitoring may encounter the challenge of balancing performance overhead with monitoring accuracy. For example, overly frequent status collection can impact actual data processing performance, while excessively long collection intervals may prevent timely detection of execution anomalies. To address this, the data processing system employs an intelligent sampling strategy: First, a state prediction model for the execution phase is established, analyzing characteristic patterns at different stages based on historical execution data. The sampling frequency is then dynamically adjusted based on the prediction model, increasing the sampling frequency at critical points or when signs of anomalies appear, and appropriately reducing it during stable execution. The system also implements a caching mechanism to temporarily store recent state data, avoiding frequent persistence operations. When a potential execution anomaly is detected, the system automatically increases the monitoring level and collects more detailed diagnostic information, providing a basis for subsequent problem analysis and resolution.

[0127] S612: When it is determined based on the execution status information that the processing progress of the current execution group is stagnant, split the unprocessed data of the current execution group into multiple sub-execution groups.

[0128] Among them, the execution status information represents the runtime status data of the group to be executed; the processing progress stagnation means that the data processing rate within the preset time window is significantly lower than expected or there is no progress at all; the current execution group represents the data set being processed; the unprocessed data represents the data records that have not yet completed the calculation; the sub-execution group represents multiple smaller execution units formed after splitting the large execution group.

[0129] After detecting an execution anomaly, the data processing system needs to make timely task adjustments. Specifically, the data processing system first determines whether the current execution group has experienced a processing stagnation based on the execution status information. Judgment criteria include: whether the data processing rate in the recent period is lower than a certain percentage of the historical average (such as 50%), whether no new data processing has been completed for multiple consecutive sampling periods, and whether system resource usage is in an abnormal state. When it is confirmed that the execution group's processing is stagnant, the system will immediately suspend the execution group's processing and count the remaining unprocessed data. The system will then initiate a task splitting mechanism to divide the unprocessed data into multiple smaller sub-execution groups in preparation for subsequent rescheduling.

[0130] In some embodiments, task splitting can be achieved in a variety of ways: Optionally, the data processing system can adopt an adaptive splitting strategy, and the specific steps include: analyzing the cause of the stagnation to distinguish whether it is caused by data characteristics or resource competition; determining the appropriate sub-execution group size based on the current resource status of the system; considering the correlation of the data to divide the boundaries; allocating independent resource quotas to each sub-execution group; and generating a new execution plan. Optionally, the data processing system can also adopt a progressive splitting method to gradually adjust the task granularity through multiple rounds of splitting, and the specific steps include: initially splitting the task into larger sub-tasks; monitoring the execution effect of the sub-tasks; performing secondary splitting when necessary; and continuously optimizing the task division. It is understandable that other methods can also be used to achieve task splitting, which are not limited here.

[0131] During the actual execution process, task splitting may encounter the problem of handling data dependencies. For example, when there are complex associations between the data to be processed, simply dividing the data equally may undermine the integrity of the data processing. To address this situation, the data processing system adopts a semantic-aware splitting strategy: first, the dependencies between data records are analyzed to build a data dependency graph; then, tasks are divided according to the structural characteristics of the dependency graph to ensure that data with dependencies are assigned to the same sub-execution group; at the same time, the physical storage location of the data is taken into consideration to minimize cross-node data access. The system also maintains a splitting effect evaluation mechanism to record the execution effects of different splitting schemes, which is used to optimize subsequent task splitting strategies for similar scenarios. When it is found that the subtasks after splitting still have execution anomalies, the system will trigger dynamic adjustments to the splitting strategy, including adjusting the splitting granularity, updating dependency analysis rules, etc.

[0132] S613: Insert multiple sub-execution groups into the execution priority sequence to regenerate the execution priority sequence.

[0133] Among them, multiple sub-execution groups represent several smaller execution units obtained by splitting the stalled tasks; the execution priority sequence represents a queue of tasks to be executed sorted by data size; the insertion operation represents the process of adding a new sub-execution group to the existing priority sequence; and the regeneration operation represents the operation of reordering the entire sequence after adding the new sub-execution group.

[0134] After completing task splitting, the data processing system needs to rationally arrange the execution order of the sub-execution groups. Specifically, the data processing system first evaluates the characteristics of each sub-execution group, including information on dimensions such as data size, estimated execution time, and resource requirements. These sub-execution groups are then inserted into the existing execution priority sequence, with the insertion position determined by the data size of the sub-execution group. After the insertion is complete, the system reorders the entire priority sequence to ensure that the order of data size is maintained from largest to smallest. This process requires ensuring the atomicity of the operation to avoid priority judgment errors or duplicate task execution during the sequence update process.

[0135] In some embodiments, priority management of sub-execution groups can be achieved in a variety of ways: Optionally, the data processing system can adopt a multi-dimensional priority calculation method, and the specific steps include: building a comprehensive scoring model that includes factors such as data volume, complexity, and dependencies; calculating a priority score for each sub-execution group; determining the insertion position based on the score; handling conflicts when priorities are similar; and maintaining the orderliness of the priority sequence. Optionally, the data processing system can also adopt a dynamic priority adjustment method to update the task priority in real time based on execution feedback, and the specific steps include: monitoring the execution effect of the sub-execution group; analyzing performance; dynamically adjusting priority weights; and reorganizing the execution sequence. It is understandable that other methods can also be used to achieve priority sequence management, which is not limited here.

[0136] S614: Calculate the to-be-executed group with the highest priority in the execution priority sequence to generate an execution result.

[0137] Referring to step S406 , the data processing system generates an execution result.

[0138] S615: After all N to-be-executed groups have been executed, generate query results based on the corresponding execution results.

[0139] Referring to step S407 , the data processing system generates a query result.

[0140] In some embodiments, the data processing system will perform performance scoring, that is, the data processing system will collect execution indicator information of each group to be executed; the execution indicator information includes actual execution time, resource usage and data processing rate; based on the execution indicator information, the performance score of this query is calculated.

[0141] Among them, the execution indicator information represents the quantitative measurement data of the operation status of the execution group; the actual execution time refers to the time interval from the start of processing to the completion of calculation; the resource usage represents the total amount of computing resources consumed during the execution process; the data processing rate represents the number of data records processed per unit time; and the performance score represents the query performance measurement value obtained by comprehensive calculation based on multiple execution indicators.

[0142] During the query execution process, the data processing system needs to conduct real-time evaluation of the execution effect. Specifically, the data processing system will establish an independent performance monitoring mechanism for each group to be executed and continuously collect key execution indicators. In terms of execution time, the system will record the task start time, data loading completion time, time distribution of the calculation phase, and final end time. For resource usage, the system will count indicators such as CPU time, memory usage peak, and I / O operation volume. In terms of data processing, the system will track and record the actual data throughput, including the number of records processed per second, data scanning speed, calculation speed, etc. Based on the collected execution indicator information, the data processing system calculates the overall performance score of this query through a preset scoring model. This score reflects the efficiency of query execution and the rationality of resource utilization.

[0143] In some embodiments, the execution effect evaluation can be achieved in a variety of ways: Optionally, the data processing system can adopt a scoring method based on historical benchmarks, by comparing with the historical execution data of similar queries, calculating the performance deviation and performing normalization to generate the final score. Optionally, the data processing system can also adopt a dynamic weight scoring method, adaptively adjusting the weight ratio of different indicators in the score according to the query type and the characteristics of the execution environment. It is understandable that other methods can also be used to achieve the execution effect evaluation. In addition, the system needs to consider the differences in scoring standards for different scales of data and different types of queries, as well as the impact of changes in the execution environment on the score.

[0144] In some embodiments, the data processing system will adjust parameters based on the score, that is, when the performance score is lower than the preset performance threshold, the data processing system will calculate the resource utilization of each to-be-executed group based on the resource usage in the execution indicator information; adjust the preset grouping coefficient based on the resource utilization, and adjust the minimum number of scan rows based on the data processing rate in the execution indicator information.

[0145] Among them, the performance score represents a comprehensive measure of the query execution quality; the preset performance threshold represents the minimum performance standard that the system expects to achieve; the resource utilization rate represents the actual use efficiency of computing resources; the preset grouping coefficient represents the adjustment parameter used to calculate the number of execution groups; the minimum number of scan rows represents the minimum amount of data that a single execution group should process; and the resource usage represents the total amount of system resources consumed during the execution process.

[0146] After completing the performance evaluation, the data processing system needs to optimize and adjust for performance deficiencies. Specifically, the data processing system first compares the calculated performance score with the preset performance threshold, and triggers the optimization process when the score is lower than the threshold. The system analyzes the resource usage of each group to be executed and calculates the actual utilization of key resources such as CPU and memory. By comparing the differences in resource utilization between different execution groups, the system can identify whether the resource allocation is reasonable. Based on the analysis results of resource utilization, the system will adjust the preset grouping coefficient accordingly. When resource utilization is generally low, the grouping coefficient will be increased to improve parallelism, and when resource competition is severe, the grouping coefficient will be reduced. At the same time, the system will dynamically adjust the minimum number of scan rows based on the data processing rate observed during execution to ensure that each execution group can obtain the appropriate amount of data processing.

[0147] In some embodiments, parameter optimization and adjustment can be achieved in a variety of ways: Optionally, the data processing system can adopt a progressive tuning method, by adjusting the parameters in small steps and observing the execution effect, gradually finding the optimal parameter configuration. Optionally, the data processing system can also adopt a model-based optimization method, by establishing a relationship model between parameters and performance, and predicting the execution effect under different parameter values. It is understandable that other methods can also be used to achieve parameter optimization and adjustment. In addition, the system needs to consider the stability of parameter adjustment to avoid frequent large-scale adjustments that lead to unstable execution behavior.

[0148] During actual execution, parameter adjustments may encounter conflicting optimization objectives. For example, increasing parallelism may increase processing speed but reduce resource utilization, while reducing the minimum number of scan rows may improve load balancing but increase scheduling overhead. To address this issue, the data processing system employs a multi-objective balancing optimization strategy: first, a comprehensive evaluation model is constructed that incorporates multiple performance metrics and assigns weights to different optimization objectives. Then, the impact of parameter adjustments on each metric is analyzed based on historical execution data. Finally, by solving the multi-objective optimization problem, a parameter configuration that strikes a balance between the various metrics is identified. The system also implements a rollback mechanism for parameter adjustments, allowing for a quick reversion to the previous configuration if performance after adjustment fails to achieve the expected improvement. Furthermore, the system continuously accumulates experience with parameter adjustments and establishes a library of optimization strategies tailored to different query scenarios, improving the accuracy and efficiency of parameter adjustments.

[0149] In the embodiments of the present application, due to the adoption of dynamic grouping strategies, priority scheduling mechanisms and execution status monitoring mechanisms, the execution plan can be adaptively adjusted according to system resources and data characteristics, effectively solving the problems of fierce resource competition, low cache utilization and unbalanced load in traditional parallel execution methods, thereby achieving a significant improvement in distributed database query performance and optimization of resource utilization efficiency.

[0150] The data processing system in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 7 , is a schematic diagram of a physical device structure of a data processing system in an embodiment of the present application.

[0151] It should be noted that Figure 7 The structure of the data processing system shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0152] like Figure 7 As shown, the data processing system includes a CPU 701, which can perform various appropriate actions and processes according to the programs stored in the ROM 702 or the programs loaded from the storage unit 708 into the RAM 703, such as executing the methods described in the above embodiments. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An I / O interface 705 is also connected to the bus 704.

[0153] The following components are connected to the I / O interface 705: an input section 706 including an audio input device, push button switches, and the like; an output section 707 including a liquid crystal display (LCD), an audio output device, indicator lights, and the like; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read from the removable media can be installed in the storage section 708 as needed.

[0154] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709 and / or installed from the removable medium 711. When the computer program is executed by the CPU 701, the various functions defined in the present invention are performed.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0156] Specifically, the data processing system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the distributed database query method provided by the above embodiment is implemented.

[0157] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the data processing system described in the above embodiments, or may exist independently and not incorporated into the data processing system. The storage medium carries one or more computer programs, which, when executed by a processor of the data processing system, enable the data processing system to implement the distributed database query method provided in the above embodiments.

Claims

1. A distributed database query method, characterized in that: Applied to a data processing system, the method comprises: Receive query requests and parse them to obtain the calculation operation type and the corresponding data distribution key; determining an input table corresponding to the query request, and generating a local execution identifier when the input table contains the data distribution key; When the local execution identifier is detected, the number of execution groups N is calculated based on the number of system CPU cores, the minimum number of scan lines and the number of physical groups; N is a positive integer; the step of calculating the number of execution groups N based on the number of system CPU cores, the minimum number of scan lines and the number of physical groups when the local execution identifier is detected specifically includes: when the local execution identifier is detected, the number of system CPU cores is obtained, and the corresponding single-thread grouping expected value is determined; the single-thread grouping expected value is half of the number of system CPU cores; a first candidate value is determined based on the product of the single-thread grouping expected value and a preset grouping coefficient; a second candidate value is determined based on the ratio of the average number of rows of the scan node to the minimum number of scan lines; the smaller value of the first candidate value and the second candidate value is determined as the maximum number of groups; the larger value of the maximum number of groups and the single-thread grouping expected value is determined as the expected number of groups; the smaller value of the expected number of groups and the number of physical groups is determined as the number of execution groups N; the preset grouping coefficient is greater than 1; Perform group calculation based on the distributed hash value of the data in the input table to generate N groups to be executed; Obtaining the data volume of each of the to-be-executed groups, and generating an execution priority sequence sorted in descending order according to the data volume; Calculate the to-be-executed group with the highest priority in the execution priority sequence to generate an execution result; After the N to-be-executed groups are all executed, a query result is generated based on the corresponding execution results.

2. The method according to claim 1, characterized in that The computing operation type includes an aggregation operation and a join operation; the step of determining the input table corresponding to the query request and generating a local execution identifier when the input table includes the data distribution key specifically includes: Determining an input table corresponding to the query request; When the computing operation type is a join operation, determining whether the input tables belong to the same collaborative distribution group; If so, when it is determined that the input tables are the same table and the data distribution key includes a join condition column, a local execution identifier is generated.

3. The method according to claim 1, characterized in that After the step of receiving and parsing the query request to obtain the calculation operation type and the corresponding data distribution key, the method further includes: Determining an input table corresponding to the query request, and when the input table does not contain the data distribution key, obtaining resource usage status of all computing nodes; Calculating the available resource score of each computing node according to the resource usage status; Sort the computing nodes based on the available resource scores, and select M computing nodes with the highest scores as candidate execution nodes; where M is a positive integer; The data records of the input table are distributed to the M candidate execution nodes.

4. The method according to claim 1, wherein After the N to-be-executed groups are all executed, and after the step of generating query results based on the corresponding execution results, the method further includes: Collecting execution indicator information of each of the to-be-executed groups; the execution indicator information includes actual execution time, resource usage, and data processing rate; Based on the execution index information, a performance score for this query is calculated.

5. The method according to claim 4, characterized in that After the step of calculating the performance score of the query based on the execution index information, the method further includes: When the performance score is lower than a preset performance threshold, calculating the resource utilization rate of each of the to-be-executed groups based on the resource usage in the execution indicator information; The preset grouping coefficient is adjusted based on the resource utilization, and the minimum number of scan lines is adjusted based on the data processing rate in the execution indicator information.

6. The method according to claim 1, characterized in that After the step of obtaining the data volume of each to-be-executed group and generating an execution priority sequence sorted in descending order according to the data volume, the method further includes: Obtaining execution status information of the to-be-executed group; When it is determined based on the execution status information that the processing progress of the current execution group is stagnant, splitting the unprocessed data of the current execution group into multiple sub-execution groups; The plurality of sub-execution groups are inserted into the execution priority sequence to regenerate the execution priority sequence.

7. A data processing system, characterized in that: The data processing system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the data processing system to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a data processing system, the data processing system is caused to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product is run on a data processing system, the data processing system is caused to perform the method according to any one of claims 1 to 6.

Images