Task concurrency dynamic adjustment method and device, equipment and medium

Abstract

The invention relates to the technical field of task parallel execution and resource scheduling, and discloses a task concurrency dynamic adjustment method, device, equipment and medium, comprising: acquiring a target execution plan fragment corresponding to a task request and traversing operator nodes thereof, identifying a data scanning operator and reading a predicted scanning data volume, accumulating the predicted scanning data volume to obtain a total scanning data volume, obtaining the number of available processor cores and a reference data volume parameter, and determining a target concurrency numerical value according to a numerical relationship between the total scanning data volume and the reference data volume parameter in combination with the number of the available processor cores, and executing the target execution plan fragment according to the target concurrency numerical value. According to the method, the concurrency is determined by summarizing the data scales of the execution plan segments and combining the processor resources, so that the parallel execution scale is adaptively adjusted along with the load change, the resource utilization rationality is improved, and the performance loss caused by configuration mismatch is reduced.

Description

Technical Field

[0001] This invention relates to the field of parallel task execution and resource scheduling technology, and in particular to a method, apparatus, device and medium for dynamically adjusting task concurrency. Background Technology

[0002] After receiving a query request, a database system typically needs to convert the declarative query statement into an executable execution plan. The execution plan describes the organizational structure of various operators during the query process and the data flow paths between operators, presented in a tree structure. During the execution plan generation phase, the query optimizer comprehensively considers factors such as table structure, index information, and data distribution, evaluates different execution paths, and generates execution plans. However, this phase primarily focuses on execution path selection and does not establish a control mechanism directly related to the number of parallel instances in subsequent execution phases and the specific task load.

[0003] In a distributed execution environment, execution plans are often broken down into multiple execution plan fragments, with data transfer between fragments achieved through data exchange nodes. Execution plan fragments can be instantiated into multiple execution plan fragment instances, allowing for parallel processing of data shards across different computing nodes. In existing technologies, the number of execution plan fragment instances is typically determined by pre-configured concurrency parameters. These parameters are determined before task startup and remain unchanged during execution, making it difficult to dynamically adjust the number of instances based on the internal operator structure of the execution plan fragment and the expected scale of the scanned data.

[0004] While database systems provide session variables to control connection-level execution behavior, existing session variables are primarily used for uniformly configuring execution strategies or resource limits, lacking the ability to determine concurrency levels based on the load differences of different execution plan segments within a single query request. Without a concurrency determination mechanism that combines the expected amount of scanned data with the number of available processor cores, existing technologies struggle to create parallel execution configurations that match the load scale during the execution plan segment scheduling phase, thus limiting the execution engine's rational use of computing resources under different load conditions. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for dynamically adjusting task concurrency, aiming to solve the technical problem in the prior art where the number of parallel execution instances depends on a fixed configuration and cannot dynamically adjust the concurrency based on the data size of the execution plan fragment and processor resources.

[0006] To achieve the above objectives, the present invention provides a method for dynamically adjusting task concurrency, comprising: Obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment; Identify the data scan operators in the traversed operator nodes and read the expected scan data volume recorded in the data scan operators; The total scan data volume corresponding to the target execution plan segment is obtained by summing up the expected scan data volume corresponding to all data scan operators in the target execution plan segment. Obtain the number of available processor cores and the preset baseline data volume parameters; The target concurrency value of the target execution plan segment is determined based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; Execute the target execution plan segment based on the target concurrency value.

[0007] Furthermore, to achieve the above objectives, the present invention provides a task concurrency dynamic adjustment device, comprising: The execution plan parsing and traversal module is used to obtain the target execution plan fragment corresponding to the task request and traverse the operator nodes contained in the target execution plan fragment; The scanning operator identification module is used to identify the data scanning operators in the traversed operator nodes and read the expected amount of scanned data recorded in the data scanning operators; The scan data volume summation module is used to accumulate the expected scan data volume corresponding to all data scan operators in the target execution plan segment to obtain the total scan data volume corresponding to the target execution plan segment. The concurrent computing parameter acquisition module is used to obtain the number of available processor cores and the preset baseline data volume parameters; The concurrency decision calculation module is used to determine the target concurrency value of the target execution plan segment based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; The parallel scheduling module for execution plans is used to execute the target execution plan segment according to the target concurrency value.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a task concurrency dynamic adjustment program stored in the memory and executable on the processor, wherein when the task concurrency dynamic adjustment program is executed by the processor, it implements the steps of the task concurrency dynamic adjustment method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a task concurrency dynamic adjustment program, wherein when the task concurrency dynamic adjustment program is executed by a processor, it implements the steps of the task concurrency dynamic adjustment method as described above.

[0010] Beneficial Effects: This invention relates to the field of task parallel execution and resource scheduling technology, and discloses a method, apparatus, device, and medium for dynamically adjusting task concurrency. The method includes: acquiring a target execution plan segment corresponding to a task request and traversing the operator nodes contained in the target execution plan segment; identifying data scan operators in the operator nodes and reading the expected scan data volume recorded in the data scan operators; summing the expected scan data volumes corresponding to all data scan operators in the target execution plan segment to obtain the total scan data volume corresponding to the target execution plan segment; acquiring the number of available processor cores and a preset baseline data volume parameter; determining the target concurrency value of the target execution plan segment based on the numerical relationship between the total scan data volume and the baseline data volume parameter, combined with the number of available processor cores; and executing the target execution plan segment according to the target concurrency value. This invention summarizes the expected scan data volumes of data scan operators at the execution plan segment level to form the total scan data volume, and introduces the number of available processor cores and the baseline data volume parameter to jointly determine the target concurrency value. This allows the target concurrency value to take effect in conjunction with changes in task load and hardware resources, improving the resource matching degree of parallel processing. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for the dynamic adjustment method of task concurrency in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the task concurrency dynamic adjustment method of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the task concurrency dynamic adjustment device of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The method for dynamically adjusting task concurrency provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the target execution plan fragment corresponding to the task request from the client, traverse the operator nodes contained in the target execution plan fragment, identify the data scan operators in the operator nodes, read the expected scan data volume recorded in the data scan operators, accumulate the expected scan data volumes corresponding to all data scan operators in the target execution plan fragment to obtain the total scan data volume corresponding to the target execution plan fragment, obtain the number of available processor cores and the preset baseline data volume parameter, determine the target concurrency value of the target execution plan fragment based on the numerical relationship between the total scan data volume and the baseline data volume parameter, and combine it with the number of available processor cores, and execute the target execution plan fragment according to the target concurrency value. This invention summarizes the expected scan data volume of data scan operators at the execution plan fragment level to form the total scan data volume, and introduces the number of available processor cores and the baseline data volume parameter to jointly determine the target concurrency value, so that the target concurrency value takes effect in conjunction with changes in task load and hardware resources, improving the resource matching degree of parallel processing. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the task concurrency dynamic adjustment method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the task concurrency dynamic adjustment method proposed in this invention includes the following steps: S10, obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment; In this embodiment, the task request enters the execution environment in the form of a structured query instruction. This task request only describes the desired data processing result and does not directly contain information that can be used for execution scheduling or structural analysis. To ensure that subsequent processing can be built upon a clear execution structure, the task request needs to be converted into an execution representation that can be parsed by the execution engine. Based on the data objects, operational relationships, and dependency constraints involved in the task request, the system generates a corresponding execution plan expression structure, within which multiple data processing units with independent execution boundaries are formed.

[0016] In the execution plan representation structure, the target execution plan fragment represents the execution unit that needs to be analyzed and processed. This execution plan fragment consists of multiple operator nodes, which describe the specific data processing behaviors and their connection relationships. The operator nodes form a directed structure through data flow relationships, enabling data to be processed step by step along a predefined path.

[0017] After determining the target execution plan segment, it is necessary to perform a complete access to the operator nodes within that segment to obtain the full picture of the execution structure. To this end, the system accesses each operator node one by one based on the node relationships maintained within the execution plan segment, ensuring that all operator nodes in the target execution plan segment are covered. This access process is used to establish an internal understanding of the execution structure; it does not involve modifying or executing operator nodes, but only serves to provide a structural foundation for subsequent processing.

[0018] This embodiment maps task requests to target execution plan fragments and performs complete access to the operator nodes within the fragments, enabling the execution structure to be accurately identified and understood in the early stages of processing. This avoids subsequent processing in the absence of execution structure information, thereby improving the completeness and consistency of execution structure analysis.

[0019] S20, identify the data scanning operators in the traversed operator nodes, and read the expected scan data volume recorded in the data scanning operators; In this embodiment, after accessing the operator nodes within the target execution plan segment, it is necessary to distinguish the node type responsible for data reading from the accessed operator nodes. Operator nodes have clearly defined functions within the execution structure, with different operator nodes corresponding to different data processing behaviors. Among them, the data scanning operator is used to read raw data from the basic data storage and is the main source of data input in the execution structure.

[0020] To achieve accurate identification, the system determines the visited operator nodes based on the type description information carried by the operator nodes themselves. This type description information is injected into the operator nodes during the execution plan generation phase to identify the processing category corresponding to the operator node. By comparing the type description information of the operator nodes with the predefined range of data scanning operator types, the set of nodes responsible for data scanning functions can be determined from multiple operator nodes.

[0021] After confirming the data scan operator, it is necessary to read information related to the data size from within the operator. The data scan operator maintains statistical information generated during the execution plan generation phase, which reflects the execution engine's estimate of the data size. By accessing the data size field in this statistical information, the expected scan data volume of the corresponding data scan operator record can be obtained. This data volume describes the expected data size that the operator needs to process during execution.

[0022] This embodiment distinguishes data scanning operators at the operator node level and reads their expected scan data volume, enabling the data input scale in the execution structure to be accurately identified, thus providing a reliable data foundation for subsequent processing around the execution load.

[0023] S30, sum up the expected scan data volume corresponding to all data scan operators in the target execution plan segment to obtain the total scan data volume corresponding to the target execution plan segment; In this embodiment, after identifying the data scanning operators and obtaining the expected scan data volume recorded by each data scanning operator, it is necessary to uniformly summarize these scattered data scale information to form an aggregated result that can characterize the overall data input scale of the target execution plan segment. Since an execution plan segment may contain multiple independent data scanning operators, each corresponding to a different data source or data fragment, its expected scan data volume only reflects the local input scale and cannot be directly used to characterize the overall load.

[0024] To address this, an accumulation process for data aggregation is constructed during execution, sequentially merging multiple anticipated scan data volumes. This merging process adds the data volumes corresponding to each data scan operator one by one in a numerical addition manner, enabling the aggregation of each local scan scale within the same numerical space. In this way, data scale information distributed across different operator nodes can be converted into a single numerical value to represent the overall data scale that the target execution plan segment is expected to process during the execution phase.

[0025] The final total scan data volume is used to reflect the overall data reading pressure of the target execution plan segment. This value comes from the statistical results of each data scan operator, maintaining the integrity and consistency of the underlying data scale estimation.

[0026] This embodiment sums up the expected scan data volume of multiple data scan operators in a unified manner, so that the data scale of the target execution plan segment can be expressed in a single numerical form, providing a clear and stable data basis for subsequent processing around the overall execution load.

[0027] S40, obtain the number of available processor cores and the preset baseline data volume parameters; In this embodiment, after determining the overall scan data scale of the target execution plan fragment, it is also necessary to obtain basic parameters related to execution resources and concurrency control to characterize the parallel processing capabilities that the current operating environment can support and the reference scale for concurrency calculation. Specifically, the number of available processor cores reflects the upper limit of schedulable computing resources in the current execution environment, while the baseline data volume parameter defines the data processing scale corresponding to a unit of concurrent instances; both are derived from the system operating environment and session-level configuration, respectively.

[0028] The number of available processor cores originates from the actual allocation of computing resources to the running node, typically exposed to the database execution engine by the operating system or runtime environment. This value reflects the number of logical processors that can be scheduled by the execution engine during the current task execution, directly constraining the maximum number of subsequent parallel execution instances.

[0029] The baseline data volume parameter describes the reference value of the data scale used in the concurrency calculation process. Its value is used to measure the load baseline of a single concurrent execution unit at the data processing level. This parameter usually exists as a session-level variable to support differentiated control of concurrency strategies for different connections and different tasks.

[0030] By acquiring both types of parameters simultaneously, execution resource conditions and data scale benchmarks can be incorporated into a unified concurrency control computing foundation without relying on fixed configurations.

[0031] This embodiment obtains both the processor's available computing power and the data scale baseline parameters simultaneously, so that subsequent processing related to concurrency can be constrained by both execution resource conditions and data processing scale, avoiding concurrency configuration deviations caused by relying on only a single dimension parameter.

[0032] S50, determine the target concurrency value of the target execution plan segment based on the numerical relationship between the total scan data volume, the baseline data volume parameters, and the number of available processor cores; In this embodiment, after obtaining the overall scan data scale, concurrent computing baseline, and schedulable processor resources, it is necessary to quantize and map the data scale to computing power to generate a target concurrency value for controlling the scale of parallel execution. This concurrency value is used to express the number of parallel instances suitable for use in the execution plan segment under the current execution environment.

[0033] The target concurrency value is determined based on the numerical relationship between the scanned data volume and the baseline data volume parameter. By performing numerical calculations on both, the abstract data scale is transformed into an integer result that can be used for concurrency control. This value reflects the number of parallel execution units that theoretically need to be started under the current data scale conditions.

[0034] After obtaining the initial concurrency value, it is also necessary to combine it with the number of available processor cores for constraint processing to avoid the number of concurrent instances exceeding the computing capacity that the runtime environment can support, thereby establishing a balance between data processing needs and resource conditions.

[0035] With this calculation method, concurrency no longer depends on fixed parameters, but changes dynamically with the data size of the execution plan fragments and the runtime resources.

[0036] In one implementation, numerical operations are performed on the total scan data volume and the baseline data volume parameters to generate an intermediate value that reflects the multiple relationship of the data size, and the value is then integerized to obtain the preliminary concurrency.

[0037] In another implementation, the initial concurrency level is compared with the number of currently available processor cores. When the initial concurrency level exceeds the processor resource limit, the number of processor cores is used as the concurrency control result.

[0038] It can also introduce minimum concurrency limits or resource scaling strategies for different deployment environments to correct the boundaries of the calculation results, so as to adapt to light load or resource-constrained scenarios.

[0039] This embodiment incorporates the scale of scanned data and processor resource conditions into the concurrency calculation process, enabling the scale of parallel execution to be adjusted according to changes in task load, thus avoiding resource waste or decreased execution efficiency caused by excessive or insufficient concurrency.

[0040] S60, Execute the target execution plan segment according to the target concurrency value.

[0041] In this embodiment, after obtaining the target concurrency value, it needs to be transformed into parallel control behavior at the execution level, so that the execution plan segments run at the corresponding concurrency scale. The target concurrency value is used to constrain the number of parallel execution plan segments instantiated and scheduled during the runtime phase, thereby directly affecting the task partitioning method and resource usage pattern during the execution process.

[0042] During the execution phase, the execution plan fragment does not run as a single entity, but rather is created by copying or instantiating it into multiple independently schedulable execution units. The target concurrency value serves as the basis for the number of instantiations, limiting the number of execution units created so that each execution unit undertakes a portion of the data processing tasks.

[0043] During execution, each execution unit runs in parallel on the computing node, following a unified execution logic but processing different data ranges, and completing the overall processing task corresponding to the execution plan segment through concurrent collaboration.

[0044] In this way, the concurrency value is transformed from a calculation result into direct control over the execution behavior, forming a closed loop from resource assessment to actual operation.

[0045] In one implementation, a corresponding number of execution plan fragment instances are created based on the target concurrency value, and these instances are distributed to nodes with computing power for parallel scheduling and execution.

[0046] In another implementation, execution plan fragment instances are mapped to multiple parallel execution tasks, which are run in parallel by the execution engine within a single node through a multi-core scheduling mechanism.

[0047] It can also bind concurrent instances to specific computing resource pools to adapt to multi-node deployment or resource isolation scenarios, depending on the different operating environments.

[0048] This embodiment uses the target concurrency value directly to control the parallel execution scale of the execution plan segments, so that the execution process can be consistent with the expected concurrency control results, thereby improving the controllability of resource utilization and the stability of execution efficiency.

[0049] In one embodiment, step S10 includes: S101, Receive the task request submitted by the client, call the query optimizer to process the task request, and generate a global execution plan tree; S102, based on the data exchange nodes in the global execution plan tree, perform a structure cutting operation on the global execution plan tree to generate multiple independent execution plan fragments, and determine the currently scheduled execution plan fragment as the target execution plan fragment; S103, obtain the root node of the target execution plan segment, and initialize the tree structure traverser for the target execution plan segment; S104, using the tree structure traverser, visit each operator node contained in the target execution plan segment one by one according to the recursive path from the root node to the leaf node.

[0050] In this embodiment, the target execution plan fragment corresponding to the task request needs to be clearly located and a traversable structural entry point established before execution to support subsequent access to each operator node. The task request, as the trigger source, carries the query or task expression to be executed. After receiving the task request submitted by the client, the query optimizer processes the task request to obtain an executable, structured result. This processing addresses the semantic constraints, access objects, connection relationships, and computational logic in the task request, generating a global execution plan tree. The global execution plan tree organizes operator nodes in a tree structure. The parent-child relationship between nodes expresses the flow and dependency of data between operators. The root represents the final output position, and the leaves represent the data entry positions. The tree structure provides a unique structural basis for subsequent cutting and traversal.

[0051] The global execution plan tree needs to be further decomposed into multiple independent execution plan fragments to ensure that execution units have schedulable boundaries. Data exchange nodes act as structural markers for fragment boundaries, corresponding to the data transfer locations across fragments and forming clear dividing points within the tree structure. When performing structural cutting operations based on the data exchange nodes in the global execution plan tree, the cutting action splits the original tree into multiple independent subtree structures with the data exchange nodes as boundaries. Each subtree structure is defined as an execution plan fragment. The dependencies between multiple execution plan fragments are implicitly expressed by the data exchange nodes, and these dependencies are used to limit the data input / output order of upstream and downstream fragments. The currently scheduled execution plan fragment is selected from among multiple execution plan fragments. The selection criteria can come from the ready state, dependency satisfaction state, or resource allocation state in the scheduling queue. The selection result forms the target execution plan fragment, thus limiting the subsequent traversal range to within a single fragment and avoiding unclear boundaries and increased costs caused by traversing the global scope.

[0052] The target execution plan fragment requires a starting anchor point for traversal, with the root node providing a unique entry point. The root node corresponds to the top-level operator node within the target execution plan fragment, holding references or connections to lower-level operator nodes and covering reachable paths to all operator nodes within the fragment. When initializing the tree structure traverser for the target execution plan fragment, the tree structure traverser handles access sequence control, node access callbacks, node deduplication marking, and recursion stack management. Once the tree structure traverser is bound to the target execution plan fragment, the traversal scope is limited to the set of nodes within the target execution plan fragment. The traversal process can proceed stably based on node references and can carry context information for cross-layer propagation during access.

[0053] When accessing operator nodes using a tree traversal, the recursive path specifies the access direction and termination condition. The recursive path expands from the root node to the leaf nodes, descending layer by layer along each parent-child reference chain until a leaf node no longer points to a lower-level node. Accessing each node sequentially means performing an access action once for every reachable operator node in the target execution plan segment. Access actions can include reading node objects, node types, node attributes, and node relationships. To avoid loops caused by repeated accesses or multiple entries due to multi-parent node structures, the tree traversal can write node tags during access and skip marked nodes when encountered again. Through this recursive access method, all operator nodes in the target execution plan segment are completely covered, forming a set of node access sequences or node sets that can be used for subsequent processing.

[0054] This embodiment generates a global execution plan tree driven by task requests and performs structural cutting operations based on data exchange nodes, making the boundaries of execution units clear and accurately locating the target execution plan fragment. A unique entry point within the fragment is established through the root node, and the tree structure traverser is initialized, enabling the traversal process to have stable access control and context carrying capabilities. By accessing operator nodes one by one along the recursive path from the root node to the leaf node, the operator nodes within the target execution plan fragment are fully covered and the access order is controllable, thereby reducing the uncertainty of structural parsing within the fragment and improving the availability of subsequent processing based on the node set.

[0055] In one embodiment, step S20 above includes: S201, parse the operator type identifier of the currently accessed operator node, and determine whether the operator type identifier belongs to a preset set of scan operation types; S202, when the operator type identifier belongs to the scan operation type set, the operator node is confirmed to be a data scan operator; S203, access the query optimization statistics stored in the data scanning operator, the query optimization statistics being injected by the query optimizer during the execution plan generation phase; S204, extract the expected number of rows to be scanned for the base table from the query optimization statistics, and use the expected number of rows to be scanned as the expected amount of scanned data.

[0056] In this embodiment, operator nodes need to be classified during the traversal process. The classification goal is to identify the data scanning operator from all operator nodes and read the expected amount of scanned data. The currently visited operator node represents the node object visited by the tree structure traverser at a certain moment. The current visit limits the input range of the recognition action, enabling the recognition process to proceed node by node in the traversal sequence. The operator type identifier is used to describe the category attribute of the operator node. The operator type identifier can be stored in the metadata area of ​​the operator node in the form of an enumeration value, integer encoding, string code, or internal type handle. Parsing the operator type identifier of the currently visited operator node involves locating the type field of the operator node, reading the field value, and completing the normalization of the type value, so that subsequent judgments can be compared within a unified identifier domain.

[0057] A predefined set of scan operation types provides the criteria for determining data scan operators. This set is a predefined structure covering scan types that may trigger reads of underlying tables within the system. The term "predefined" signifies that the set is configured before execution and can be directly queried at runtime. The set structure can be a hash set, bitmap set, sorted array, or mapping table, ensuring that membership checks can be performed using a fixed interface. Determining whether an operator type identifier belongs to the predefined set involves inputting the operator type identifier as a search key into the set query interface and obtaining a "belongs" or "does not belong" result. This result is returned as a Boolean value or status code, providing a unique basis for subsequent confirmation actions.

[0058] When the operator type identifier belongs to the scan operation type set, the operator node is confirmed as a data scan operator. This confirmation action solidifies the judgment result into a node category conclusion, ensuring that subsequent reading actions are limited to the data scan operator set. Confirmation can be achieved by marking the operator node with a data scan operator tag, adding the operator node reference to the data scan operator list, or registering the data scan operator index in the traversal context. The purpose of the confirmation result is to establish a mapping relationship from operator nodes to data scan operators, avoiding repeated type checks in subsequent stages. The confirmation process can also carry a location entry point for scan object information, such as associating the scan descriptor reference inside the operator node with the data scan operator tag, thereby providing an object path for reading the expected scan data volume.

[0059] Reading the estimated scan data volume depends on the existence and accessibility of query optimization statistics. Query optimization statistics are a set of statistical data injected into operator nodes or their attribute structures by the query optimizer during the execution plan generation phase. They represent the query optimizer's estimate of the scan size. "Injection" means that the query optimization statistics are written into the execution plan's related data structures before execution, and can be read without recalculation during execution. The execution plan generation phase corresponds to the plan construction process before and after the generation of the global execution plan tree and the formation of execution plan fragments. Accessing the query optimization statistics stored in the data scan operator involves locating the statistics structure along the attribute pointer or metadata entry point of the data scan operator, performing structure existence checks, and reading the structure content. Existence checks handle cases where statistics are missing or fields are not initialized, while field checks handle changes in field names or offsets caused by version differences in statistics.

[0060] The estimated number of rows scanned for the base table is a key field in query optimization statistics. The base table describes the data entities directly accessed by the data scan operators, while the estimated number of rows scanned describes the estimated scan output or read scale by the query optimizer. This estimated number of rows scanned can be derived from a composite of table statistics, partition statistics, sampling statistics, or index selectivity estimates. Extracting the estimated number of rows scanned for the base table involves locating the corresponding field within the query optimization statistics structure, reading the field value, and performing numerical normalization. This normalization process includes converting field values ​​from different precision types to a unified numeric type, replacing null values ​​with default values, and pruning negative values ​​to non-negative values. This ensures that the estimated number of rows scanned can be reliably used as input for subsequent calculations. Using the estimated number of rows scanned as the estimated scan data volume reflects the assignment rules for the estimated scan data volume. The estimated scan data volume, as a scale metric at the data scan operator level, is determined as the estimated number of rows scanned, making it semantically directly correlated with the base table scan scale. Furthermore, it has a single-source field in its data structure, facilitating subsequent aggregation of the estimated scan data volumes from multiple data scan operators.

[0061] This embodiment parses the operator type identifier and completes the membership determination based on a preset set of scan operation types, making the identification of data scan operators have clear judgment criteria and enabling stable execution node by node during traversal. By confirming that the operator node is a data scan operator when the determination is successful, subsequent reading actions are limited to the confirmed node set and the overhead of repeated judgments is reduced. By accessing the query optimization statistics stored in the data scan operator and extracting the expected number of scan rows for the base table as the expected scan data volume, the expected scan data volume has a traceable data source and a unified field standard, thereby improving the consistency of the expected scan data volume reading and providing a stable input for subsequent calculations based on the expected scan data volume.

[0062] In one embodiment, step S30 above includes: S301, Construct a scan data accumulator and set the initial value of the scan data accumulator to zero; S302, for each identified data scanning operator, perform an addition operation to add the expected amount of scanned data to the scanned data accumulator to update the value of the scanned data accumulator; S303, after completing the addition operation on all identified data scanning operators, obtain the final value of the scan data accumulator; S304, the final value is used as the total scan data volume corresponding to the target execution plan segment.

[0063] In this embodiment, the total scan data volume corresponding to the target execution plan segment needs to be obtained by aggregating the expected scan data volume of all data scan operators within the target execution plan segment. The aggregation process is carried out by a scan data volume accumulator. The scan data volume accumulator is a state carrier for accumulating values. The state carrier can reside in the local context of the execution thread, the traversal context, or the segment-level context of the execution engine, thus binding the accumulation process to the target execution plan segment. Constructing the scan data volume accumulator involves three actions: allocating storage units, determining the value type, and establishing a writable entry point. The value type needs to cover the range of values ​​for the expected scan data volume and support addition operations. The storage unit can be a register variable, a heap object field, or a shared memory slot. Setting the initial value of the scan data volume accumulator to zero corresponds to the baseline setting of the accumulation semantics. The zero value is used to eliminate the interference of historical residual values ​​on this aggregation and to provide a definite initial addend for each subsequent addition operation.

[0064] The identified data scan operator is the set of nodes output from the previous identification action. This set can be presented as a list, queue, set structure, or the immediate judgment result during traversal. Performing addition operations on each identified data scan operator reflects the completeness constraint of the aggregation object; the aggregation object is limited to the identified data scan operators, avoiding the inclusion of non-scanning operators in the accumulation. The read expected scan data volume is the size estimate inherent to each data scan operator. This value needs to be normalized before entering the addition operation to ensure computational feasibility. Normalization can include null value replacement, negative value pruning, overflow detection, and precision alignment, ensuring that fields from different sources or of different types form a unified, addable value. Performing the addition operation involves retrieving the current value from the scan data volume accumulator, retrieving the expected scan data volume, calculating the sum of the two, and writing back the updated value. The write-back action writes the updated value to the scan data volume accumulator, thus incorporating the expected scan data volume of a single data scan operator into the fragment-level accumulation result.

[0065] Updating the value of the scan data accumulator requires maintaining the accumulation order and consistency. If the addition operation is executed in a single-threaded traversal context, the order is naturally determined by the traversal order, and the update of the scan data accumulator is completed serially. If parallel traversal or parallel identification causes multiple data scan operators to concurrently trigger addition operations, the scan data accumulator needs to have concurrent writable semantics. Specifically, this can be achieved through atomic addition, segmented accumulation followed by merging, or thread-local accumulator aggregation. Segmented accumulation followed by merging involves accumulating separately through multiple local scan data accumulators and merging at the end. Thread-local accumulator aggregation maintains an independent accumulated value for each execution unit and performs a one-time summation at the termination point. Both methods maintain the aggregation caliber and reduce shared write conflicts.

[0066] After completing the addition operation on all identified data scan operators, the final value of the scan data accumulator is obtained, corresponding to the termination condition determination and result reading. The termination condition needs to be consistent with the set boundary of the identified data scan operators. The determination method can be reaching the set length through iteration counting, reaching the end of the set through traversal cursor, or triggering the reading through traversal end event. Obtaining the final value involves reading the current stored value from the scan data accumulator and forming a stable output value. The stable output value can be written to the fragment-level metadata field of the target execution plan fragment, written to the input field of the concurrency calculation context, or written to a shared structure accessible to subsequent modules. The final value is used as the total scan data volume corresponding to the target execution plan fragment to reflect the binding relationship between naming and scope. The total scan data volume is defined as the final value of the scan data accumulator, ensuring that the total scan data volume is consistent with the sum of the expected scan data volumes of all data scan operators in semantics and implementation, and providing a single entry point for subsequent numerical comparisons and concurrency calculations based on the total scan data volume.

[0067] This embodiment uses a scan data accumulator to carry the aggregation state and initializes it with a zero value, providing a deterministic starting point for the accumulation process that can be executed repeatedly. By performing addition operations only on the identified data scan operators and writing back and updating the value of the scan data accumulator after each operation, the aggregation caliber is stable and the aggregation path can be continuously advanced in the traversal context. By reading the final value of the scan data accumulator after completing all addition operations and binding it to the total scan data volume corresponding to the target execution plan segment, the output result has a single source and a clear termination condition, thus providing a consistent and directly usable input value for subsequent calculation steps that depend on the total scan data volume.

[0068] In one embodiment, step S40 above includes: S401, Call the system resource query interface of the current running environment to detect the number of logical cores allocated to the service process on the current computing node, and use the number of logical cores as the number of available processor cores; S402, access the database session context associated with the task request and retrieve the list of session variables stored in the database session context; S403, search the session variable list for the configuration item used to indicate the base number of rows for dynamic concurrency, read the configuration value recorded in the configuration item, and use the configuration value as a preset baseline data volume parameter.

[0069] In this embodiment, the number of available processor cores and a preset baseline data volume parameter are used as inputs for subsequent numerical comparisons and concurrency calculations. These parameters need to be obtained from both the runtime environment and the database session side and formed into numerical forms that can be directly used in the calculations. The entry point for the resource detection action is selected by calling the system resource query interface of the current runtime environment. The system resource query interface can be exported by the operating system kernel, encapsulated by the runtime library, or provided by the container runtime environment. Its call request includes three elements: target resource category, scope, and return format. The target resource category points to processor core resources, the scope is limited to the available range allocated to the service process by the current computing node, and the return format is used to carry the logical core count. The detection of the logical core count allocated to the service process by the current computing node corresponds to a quantitative expression of resource availability. The logical core count represents the upper limit of the execution channels that the service process can schedule in parallel. The allocation semantics come from constraints such as process scheduling affinity, resource quotas, or container limitations. The detection action needs to perform validity verification after obtaining the return value. The verification includes non-empty checks, non-negative checks, and upper limit reasonableness checks to ensure that the logical core count meets the prerequisite of the available processor core count. The logical core number is bound and named as the corresponding value of the number of available processor cores. The binding action can be expressed as writing to the task execution context, writing to the computing context associated with the target execution plan fragment, or writing to the input field of the concurrency calculation module.

[0070] The acquisition of preset baseline data volume parameters begins with the task request as the index to enter the database session side. Accessing the database session context associated with the task request determines the request's ownership. When the task request is passed between the access layer and the execution layer, it carries a connection identifier, session identifier, or execution handle. These identifiers are used to locate the database session context within the session management structure. The database session context carries runtime configuration and control information related to a single connection. Its structure may include a parameter mapping table, namespace, version tag, and effective status, supporting the reading and updating of session-level configurations. Retrieving the session variable list stored in the database session context corresponds to the enumeration and query entry point for the parameter set. The session variable list is organized in key-value pairs or structured entries. Each entry contains at least a key name, type, current value, and effective domain, ensuring a definite matching target for subsequent lookup actions. The retrieval action requires completing list reachability and consistency checks. Reachability checks confirm that the session variable list has been loaded into memory or can be obtained through lazy loading. Consistency checks confirm that the currently read view matches the session identifier of the task request, avoiding parameter misuse caused by cross-connection reads.

[0071] The process involves locating the baseline parameter corresponding to the configuration item indicating the base row count for dynamic concurrency from the session variable list. The search criteria are carried by the configuration item's key name or identifier field. The key name can be a fixed string, an internal enumeration, or a hash identifier. The identifier field can contain parameter category and usage markers, thus distinguishing the base row count for dynamic concurrency from other resource control parameters. The search process handles missing and duplicate paths. For missing paths, usable values ​​can be obtained through default value parsing, inheriting from parent configurations, or falling back to the system-level configuration source. For duplicate paths, the effective entry can be selected based on priority rules, which can be defined by session-wide overriding priority, most recent update time priority, or explicit effective flag priority. The process also includes reading the configuration value recorded in the configuration item record, extracting and converting the corresponding value. Configuration values ​​may be stored as strings, integers, or long integers. After reading, type parsing, range validation, and unit consistency are required. Range validation excludes zero, negative, or abnormal configurations exceeding the allowed upper limit. Unit consistency ensures that the configuration value and the total scanned data volume are measured on the same scale. The configuration value is used as the preset baseline data volume parameter to establish the parameter placement and subsequent referenceability. The placement action is consistent with the number of available processor cores and needs to be written to the input field of the concurrency calculation context or the task-level calculation cache.

[0072] This embodiment obtains the number of logical cores through the system resource query interface and binds it to the number of available processor cores, so that the concurrency calculation can be constrained by the parallel capability actually allocated to the service process by the runtime environment. The database session context is located by the task request and the configuration item indicating the base number of rows of dynamic concurrency is found in the session variable list. The read configuration value is bound to the preset baseline data volume parameter, so that the baseline of the data volume scale comes from the session-level configuration and is consistent with the connection isolation semantics. Thus, a set of input values ​​that simultaneously cover hardware availability and session configuration baseline are formed before the concurrency calculation, reducing the sensitivity of concurrency derivation to environmental differences and parameter cross-referencing.

[0073] In one embodiment, step S50 above includes: S501, Perform a division operation between the total scan data volume and the reference data volume parameter to obtain a numerical ratio; S502, perform an up-rounding operation on the numerical ratio to generate a preliminary concurrency value. If the preliminary concurrency value is less than one, adjust the preliminary concurrency value to one. S503, compare the initial concurrency value with the number of available processor cores; S504, if the initial concurrency value is less than or equal to the number of available processor cores, then the initial concurrency value is determined as the target concurrency value; S505, if the initial concurrency value is greater than the number of available processor cores, then the number of available processor cores is determined as the target concurrency value.

[0074] In this embodiment, the total scan data volume, the baseline data volume parameter, and the number of available processor cores jointly limit the value space of the target concurrency level. The calculation process must simultaneously satisfy three constraints: load characterization, integerization, and resource limits. A division operation is performed on the total scan data volume and the baseline data volume parameter to correspond to load normalization. The total scan data volume provides a cumulative load representation of the target execution plan segment in the data scan dimension, while the baseline data volume parameter provides a reference unit for the load scale. The two form a numerical ratio under the same measurement caliber. The meaning of the numerical ratio indicates the multiple relationship between the total scan data volume and the baseline data volume parameter. A larger numerical ratio indicates a stronger demand for unit load segmentation, while a smaller numerical ratio indicates that the load is close to or below the baseline scale. The division operation can be implemented using a combination of integer division and remainder determination, or floating-point division to obtain a numerical ratio with decimals. The operation needs to handle the validity of the baseline data volume parameter value, including non-zero verification and positive value verification, to avoid division by zero or negative value propagation leading to unusable results for the target concurrency level value.

[0075] Performing a floor function on the numerical ratio corresponds to a discretization decision. Floor function mapping the numerical ratio to the smallest integer not less than the numerical ratio gives the initial concurrency value an instantiable counting semantic. The floor function can be implemented directly by a floor function or by extracting the integer part and determining the fractional part. If the fractional part exists, the integer part is incremented by one; if the fractional part is zero, the integer part remains unchanged. Generating the initial concurrency value corresponds to forming a load-driven concurrency candidate value before introducing hardware limitations. This candidate value maintains a monotonic relationship with the numerical ratio, ensuring that the initial concurrency value does not decrease as the total scan data volume increases, thus keeping the concurrency derivation consistent with load changes. If the initial concurrency value is less than one, the initial concurrency value is adjusted to one. This corresponds to the introduction of the lower bound constraint, which is used to exclude the non-executable state of zero concurrency or negative concurrency. The sources of this condition include the total amount of scanned data being zero, the numerical ratio being zero, or the result after rounding up being zero. The adjustment action fixes the initial concurrency value to one, so that the subsequent comparison and assignment process runs within a unified positive integer domain.

[0076] The initial concurrency value is compared with the number of available processor cores, which is the entry point for calculating resource availability constraints. The number of available processor cores represents the upper limit of processor core resources that the service process can occupy in parallel. The numerical comparison generates a comparison result marker, which drives subsequent branch selection. The comparison can be implemented using integer comparison instructions or comparison functions, and outputs two types of mutual exclusion relationships: less than or equal to, and greater than. If the initial concurrency value is less than or equal to the number of available processor cores, the initial concurrency value is determined as the direct adoption under the scenario where resources are not saturated, corresponding to the target concurrency value. Direct adoption ensures that the load characterization result is not reduced, and that the numerical relationship between the target concurrency value and the parameters of total scan data volume and baseline data volume is consistent, while ensuring that the target concurrency value does not exceed the number of available processor cores. If the initial concurrency value is greater than the number of available processor cores, the number of available processor cores is determined as the capping processing under the scenario where resources are saturated, corresponding to the target concurrency value. Capping processing limits the target concurrency value within the hardware availability boundary, avoiding the mismatch between the number of instances and core resources caused by the target concurrency value exceeding the number of available processor cores. The two branches cover all possible values ​​under the condition of mutual exclusion of the comparison results, so that the generation process of the target concurrency value is closed in the numerical domain. The output is always an integer greater than or equal to one and less than or equal to the number of available processor cores, and the non-decreasing trend of increasing with the total amount of scanned data is maintained before the capping.

[0077] This embodiment obtains a numerical ratio by performing a division operation on the total scan data volume and the baseline data volume parameter, and then performs an up-rounding operation on the numerical ratio to generate an initial concurrency value. A lower bound constraint is then introduced to adjust the initial concurrency value to one when it is less than one, so that the concurrency value derivation result stably falls within the instantiable positive integer domain. By comparing the initial concurrency value with the number of available processor cores and selecting the initial concurrency value or the number of available processor cores as the target concurrency value according to the comparison result, the target concurrency value simultaneously satisfies the load characterization and resource upper limit constraints, thereby maintaining the executability and boundary consistency of the concurrency value under load changes and hardware availability changes.

[0078] In one embodiment, step S60 above includes: S601, using the target concurrency value as the instantiation quantity parameter, instantiation processing is performed on the target execution plan fragment to create multiple execution plan fragment instances with the same number as the target concurrency value; S602, schedule the created multiple execution plan fragment instances to the computing nodes of the database system; S603, control the computing node to run the multiple execution plan fragment instances in parallel to complete the data processing operation of the task request.

[0079] In this embodiment, the target concurrency value serves as a control variable during the execution phase. Its meaning corresponds to the baseline for determining the number of parallel instances. The process of executing the target execution plan fragment requires converting this value into a schedulable and runnable set of instances and completing cross-node parallel startup. The target concurrency value is used as an instantiation quantity parameter to represent the mapping relationship between the quantity parameter and the instance set size. The instantiation quantity parameter enters the entry point of the instantiation process, determining the number of times the target execution plan fragment is instantiated and the upper limit of the instance object count, thus transforming concurrency control from the numerical domain to the object domain. Performing instantiation processing on the target execution plan fragment corresponds to the expansion operation from logical structure to running entity. The target execution plan fragment provides a carrier for operator node organization relationships, data exchange node association relationships, and execution control information. Instantiation processing requires generating multiple execution plan fragment instances based on this carrier. Each execution plan fragment instance retains the same structural constraints and operator node topology as the target execution plan fragment, while possessing an independent running context to support parallel execution. The creation of multiple execution plan fragment instances, each with the same number as the target concurrency level, is subject to a consistency constraint on the instance set's count. This constraint requires that the number of elements in the instance set be strictly equal to the target concurrency level. The creation process necessitates accumulating and recording instance counts or allocating indexes to prevent insufficient parallelism due to under-creation or excessive resource consumption on compute nodes due to over-creation. Instantiation can be implemented using a combination of structure copying and context reconstruction. Structure copying replicates the operator node connections of the target execution plan fragment, while context reconstruction generates independent runtime identifiers, buffer references, data channel endpoints, and scheduling metadata for each execution plan fragment instance. This ensures that different execution plan fragment instances, running in parallel on the same or different compute nodes, do not overwrite each other's runtime states.

[0080] The scheduling process involves mapping multiple execution plan fragment instances to the compute nodes of the database system, corresponding to the mapping from an instance set to a node set. This scheduling action requires determining the compute node to carry out execution for each execution plan fragment instance and establishing a binding relationship between the instance and the compute node. The compute node of the database system, as the execution carrying unit, encompasses the processing entity capable of receiving scheduling instructions and initiating execution plan fragment instances. The scheduling process needs to consider compute node availability, load status, and resource quotas to form the instance allocation result. Scheduling can be implemented based on queued dispatch or direct assignment. Queued dispatch pushes multiple execution plan fragment instances into a queue to be run according to arrival order or weight rules, and compute nodes obtain the allocated execution plan fragment instances using a pull or push mechanism. Direct assignment, after generating the allocation table, sends the runtime description information of the execution plan fragment instances to the corresponding compute nodes. The runtime description information includes at least the execution plan fragment instance identifier, the reference identifier of the target execution plan fragment, the operator node execution entry point, and data channel connection information. The scheduling process needs to ensure that the execution plan fragment instances have resolvable loading information on the compute node side. The loading information is used to complete instance registration, resource allocation and binding to running threads within the compute node, and to establish an executable state for subsequent parallel execution.

[0081] Controlling the parallel execution of multiple execution plan fragment instances on computing nodes combines execution initiation and parallel scheduling. The object of the control action is the computing node, the content of the control action is parallel execution, and the unit of execution of the control action is multiple execution plan fragment instances. Parallel execution means both the concurrent initiation of multiple execution plan fragment instances within the same computing node and the simultaneous execution between multiple computing nodes, both of which together form a concurrency scale consistent with the target concurrency value. Parallel execution requires establishing an independent running thread or running task for each execution plan fragment instance on the computing node side, and driving the data processing flow according to the operator node topology in the running thread or running task, so that the operator nodes are triggered sequentially within the execution plan fragment instance and output intermediate data. The completion of the data processing operation of the task request is the condition for the completion of parallel execution. The task request provides the trigger source and output target of data processing. The scope of the data processing operation covers the data reading, operator node processing, and result aggregation associated with the task request. During parallel execution, the computing node needs to track the execution status of multiple execution plan fragment instances. The status information includes at least a start flag, a running flag, a completion flag, and a failure flag, so as to determine that the data processing operation of the task request is completed when all execution plan fragment instances meet the completion flag. Parallel execution also requires maintaining the data collaboration relationship between execution plan fragment instances. This collaboration relationship is constrained by the data exchange nodes in the target execution plan fragment and is inherited during the instantiation and scheduling phases. The computing node side establishes data channel connections between instances based on this, enabling different execution plan fragment instances to exchange data according to the data transfer rules defined by the data exchange nodes, thereby maintaining the continuity of the overall processing link of the task request.

[0082] Example Explanation: Let's take the dynamic concurrency determination and execution process of a single Pipeline task as an example. After establishing a connection, the client sets `enable_dynamic_parallel_pipeline_task_num` to `true` in the session variable list, enabling the dynamic concurrency feature within the current database session context. Simultaneously, it configures `dynamic_parallel_row_count_base` in the session variable list to a base data volume parameter, such as ten million, to represent the data volume scale of a single parallel instance. The client then submits a task request, such as a query statement containing multi-table joins and filtering conditions. Upon receiving the task request, the system calls the query optimizer to process it. During the execution plan generation phase, the query optimizer performs semantic parsing, cost evaluation, and statistical information referencing, outputting a global execution plan tree. The global execution plan tree contains data exchange nodes. Based on these nodes, the system performs a structural segmentation operation on the global execution plan tree, dividing it into multiple independent execution plan fragments. The currently scheduled execution plan fragment is identified as the target execution plan fragment, which corresponds to the scheduling unit that will soon enter the Pipeline execution engine.

[0083] Once the target execution plan segment is determined, the system obtains the root node of the target execution plan segment and initializes a tree structure traverser for the segment. The tree structure traverser visits each operator node contained in the target execution plan segment one by one, following a recursive path from the root node to the leaf node. During the visit, each operator node is parsed, and its operator type identifier is matched against a preset set of scan operation types. The scan operation type set can include type definitions such as SeqScan, IndexScan, and ExternalScan to cover different data reading patterns. When the operator type identifier belongs to the scan operation type set, the system confirms that the operator node is a data scan operator and accesses the query optimization statistics stored in the data scan operator. These statistics are injected by the query optimizer during the execution plan generation phase and can record statistical fields such as the expected number of output rows at the operator level. The system extracts the estimated number of rows to be scanned for the base table from query optimization statistics and uses this estimated number of rows as the estimated scan data volume. For example, if the estimated number of rows to be scanned for one data scan operator is 4.5 million, then the estimated scan data volume is 4.5 million; if the estimated number of rows to be scanned for another data scan operator is 9 million, then the estimated scan data volume is 9 million. The same data reading process is performed on each identified data scan operator in the target execution plan segment, ensuring that the estimated scan data volume is continuously generated during the traversal and can be used for aggregation calculations.

[0084] During the process of traversing, identifying, and reading the expected scan data volume, the system constructs a scan data volume accumulator and sets its initial value to zero. Each time a data scan operator is identified and the expected scan data volume is read, the system immediately performs an addition operation, adding the read expected scan data volume to the scan data volume accumulator to update its value. In the example, the scan data volume accumulator becomes 4.5 million after the first addition operation, 13.5 million after the second, and if a third data scan operator is subsequently identified with an expected scan data volume of 6.5 million, the scan data volume accumulator is updated to 20 million. After completing the addition operation for all identified data scan operators, the system obtains the final value of the scan data volume accumulator and uses this final value as the total scan data volume corresponding to the target execution plan segment; in the example, the total scan data volume is 20 million.

[0085] Concurrency calculation requires combining hardware resource information and session configuration parameters. The system calls the system resource query interface of the current running environment to detect the number of logical cores allocated to the service process on the current computing node, and uses this number as the number of available processor cores. For example, if eight logical cores are detected, the system simultaneously accesses the database session context associated with the task request, retrieves the list of session variables stored in the database session context, finds the configuration item indicating the base row number for dynamic concurrency from the session variable list, and reads the configuration value recorded in the configuration item. This configuration value is used as the preset baseline data volume parameter; in this example, the preset baseline data volume parameter is ten million. After obtaining the total scan data volume, the preset baseline data volume parameter, and the number of available processor cores, the system performs a division operation on the total scan data volume and the preset baseline data volume parameter to obtain a ratio; in this example, the ratio is two. The system performs a floor operation on the ratio to generate an initial concurrency value; in this example, the initial concurrency value is two. If the initial concurrency value is less than one, it is adjusted to one to avoid non-positive values ​​for the concurrency. The system compares the initial concurrency value with the number of available processor cores. In the example, the initial concurrency value of 2 is less than or equal to the number of available processor cores of 8. Therefore, the initial concurrency value is set as the target concurrency value, which is 2. If in another task request, the total scanned data volume is 36 million while the preset baseline data volume parameter is still 10 million, the ratio is 3.6. After rounding up, the initial concurrency value is 4. The initial concurrency value, compared with the number of available processor cores of 8, still does not exceed the resource limit, so the target concurrency value is 4. If the total scanned data volume further increases, causing the initial concurrency value to exceed the number of available processor cores, the system sets the number of available processor cores as the target concurrency value, ensuring that the target concurrency value maintains a consistent resource boundary with the number of logical cores available on the computing node.

[0086] Once the target concurrency level is determined, the execution phase begins. The system uses the target concurrency level as the instantiation quantity parameter to instantiate the target execution plan fragments, creating multiple execution plan fragment instances with the same number as the target concurrency level. For example, if the target concurrency level is four, four execution plan fragment instances are created. Each execution plan fragment instance retains the operator node organization relationship and data exchange node constraints of the target execution plan fragment, while also possessing an independent runtime context to support parallel execution. The system schedules the created multiple execution plan fragment instances to the compute nodes of the database system. During the scheduling process, a binding relationship is established between the execution plan fragment instances and the compute nodes, ensuring that the execution plan fragment instances are delivered to the runnable node-side execution environment. The system controls the compute nodes to run multiple execution plan fragment instances in parallel to complete the data processing operations for the task request. Multiple instances running in parallel form concurrent execution under the Pipeline execution engine. When processing different data shards, the execution plan fragment instances can transfer and aggregate data through data exchange nodes, ultimately outputting the processing result corresponding to the task request and ending the execution chain of this task request.

[0087] This embodiment uses the target concurrency value as the instantiation quantity parameter and performs instantiation processing on the target execution plan fragments to create multiple execution plan fragment instances with the same number as the target concurrency value. This transforms concurrency control from an abstract value into a set of runnable instances while maintaining consistency in quantity. By scheduling multiple execution plan fragment instances to the computing nodes of the database system and controlling the computing nodes to run multiple execution plan fragment instances in parallel, the instance set obtains a clear execution location and forms a parallel running scale that matches the target concurrency value. This enables the execution organization and execution of data processing operations for task requests under multi-instance parallel conditions.

[0088] In one embodiment, a task concurrency dynamic adjustment device is provided, which corresponds one-to-one with the task concurrency dynamic adjustment method in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the task concurrency dynamic adjustment device of the present invention. The modules include: execution plan parsing and traversal module 10, scan operator identification module 20, scan data volume summarization module 30, concurrency calculation parameter acquisition module 40, concurrency decision calculation module 50, and execution plan parallel scheduling module 60. Detailed descriptions of each functional module are as follows: The execution plan parsing and traversal module 10 is used to obtain the target execution plan fragment corresponding to the task request and traverse the operator nodes contained in the target execution plan fragment; The scanning operator identification module 20 is used to identify the data scanning operators in the traversed operator nodes and read the expected amount of scanned data recorded in the data scanning operators; The scan data volume summarization module 30 is used to accumulate the expected scan data volume corresponding to all data scan operators in the target execution plan segment to obtain the total scan data volume corresponding to the target execution plan segment. The concurrent computing parameter acquisition module 40 is used to acquire the number of available processor cores and the preset baseline data volume parameters. The concurrency decision calculation module 50 is used to determine the target concurrency value of the target execution plan segment based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; The parallel scheduling module 60 is used to execute the target execution plan segment according to the target concurrency value.

[0089] Specific limitations regarding the dynamic task concurrency adjustment device can be found in the aforementioned limitations on the dynamic task concurrency adjustment method, and will not be repeated here. Each module in the aforementioned dynamic task concurrency adjustment device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0090] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides deterministic and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements a server-side function or step of a dynamic task concurrency adjustment method.

[0091] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a dynamic task concurrency adjustment method.

[0092] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment; Identify the data scan operators in the traversed operator nodes and read the expected scan data volume recorded in the data scan operators; The total scan data volume corresponding to the target execution plan segment is obtained by summing up the expected scan data volume corresponding to all data scan operators in the target execution plan segment. Obtain the number of available processor cores and the preset baseline data volume parameters; The target concurrency value of the target execution plan segment is determined based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; Execute the target execution plan segment based on the target concurrency value.

[0093] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment; Identify the data scan operators in the traversed operator nodes and read the expected scan data volume recorded in the data scan operators; The total scan data volume corresponding to the target execution plan segment is obtained by summing up the expected scan data volume corresponding to all data scan operators in the target execution plan segment. Obtain the number of available processor cores and the preset baseline data volume parameters; The target concurrency value of the target execution plan segment is determined based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; Execute the target execution plan segment based on the target concurrency value.

[0094] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0096] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for dynamically adjusting task concurrency, characterized in that, Includes the following steps: Obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment; Identify the data scan operators in the traversed operator nodes and read the expected scan data volume recorded in the data scan operators; The total scan data volume corresponding to the target execution plan segment is obtained by summing up the expected scan data volume corresponding to all data scan operators in the target execution plan segment. Obtain the number of available processor cores and the preset baseline data volume parameters; The target concurrency value of the target execution plan segment is determined based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; Execute the target execution plan segment based on the target concurrency value.

2. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, Obtain the target execution plan fragment corresponding to the task request, and traverse the operator nodes contained in the target execution plan fragment, including: Receive a task request submitted by the client, call the query optimizer to process the task request, and generate a global execution plan tree; Based on the data exchange nodes in the global execution plan tree, a structural cutting operation is performed on the global execution plan tree to generate multiple independent execution plan fragments, and the execution plan fragment to be scheduled is determined as the target execution plan fragment; Obtain the root node of the target execution plan fragment and initialize a tree structure traverser for the target execution plan fragment; Using the tree structure traverser, each operator node contained in the target execution plan segment is visited one by one according to the recursive path from the root node to the leaf node.

3. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, Identify the data scan operators among the traversed operator nodes, and read the expected scan data volume recorded in the data scan operators, including: Parse the operator type identifier of the currently accessed operator node and determine whether the operator type identifier belongs to a preset set of scan operation types; When the operator type identifier belongs to the set of scan operation types, the operator node is confirmed to be a data scan operator; Access the query optimization statistics stored in the data scanning operator, which are injected by the query optimizer during the execution plan generation phase; Extract the estimated number of rows to be scanned for the base table from the query optimization statistics, and use the estimated number of rows to be scanned as the estimated amount of data to be scanned.

4. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, The total scan data volume corresponding to the target execution plan segment is obtained by summing the expected scan data volume corresponding to all data scan operators in the target execution plan segment, including: Construct a scan data accumulator and set the initial value of the scan data accumulator to zero; For each identified data scanning operator, an addition operation is performed to add the expected amount of scanned data to the scanned data accumulator to update the value of the scanned data accumulator; After completing the addition operation on all identified data scan operators, the final value of the scan data accumulator is obtained; The final value is taken as the total scan data volume corresponding to the target execution plan segment.

5. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, Obtain the number of available processor cores and the preset baseline data volume parameters, including: Call the system resource query interface of the current running environment to detect the number of logical cores allocated to the service process on the current computing node, and use the number of logical cores as the number of available processor cores; Access the database session context associated with the task request and retrieve the list of session variables stored in the database session context; The configuration item used to indicate the base number of rows for dynamic concurrency is located in the session variable list, and the configuration value recorded in the configuration item is read. The configuration value is used as the preset baseline data volume parameter.

6. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, Based on the numerical relationship between the total scan data volume, the baseline data volume parameters, and the number of available processor cores, the target concurrency value of the target execution plan segment is determined, including: Perform a division operation between the total scan data volume and the reference data volume parameter to obtain the numerical ratio; The numerical ratio is rounded up to generate an initial concurrency value. If the initial concurrency value is less than one, the initial concurrency value is adjusted to one. The initial concurrency value is compared with the number of available processor cores; If the initial concurrency value is less than or equal to the number of available processor cores, then the initial concurrency value is determined as the target concurrency value; If the initial concurrency value is greater than the number of available processor cores, then the number of available processor cores is determined as the target concurrency value.

7. The method for dynamically adjusting task concurrency as described in claim 1, characterized in that, Based on the target concurrency value, execute the target execution plan segment, including: Using the target concurrency value as the instantiation quantity parameter, the target execution plan fragment is instantiated to create multiple execution plan fragment instances with the same number as the target concurrency value; The created multiple execution plan fragment instances are scheduled to the compute nodes of the database system; The computing nodes are controlled to run the multiple execution plan fragment instances in parallel to complete the data processing operations for the task request.

8. A device for dynamically adjusting task concurrency, characterized in that, The task concurrency dynamic adjustment device includes: The execution plan parsing and traversal module is used to obtain the target execution plan fragment corresponding to the task request and traverse the operator nodes contained in the target execution plan fragment; The scanning operator identification module is used to identify the data scanning operators in the traversed operator nodes and read the expected amount of scanned data recorded in the data scanning operators; The scan data volume summation module is used to accumulate the expected scan data volume corresponding to all data scan operators in the target execution plan segment to obtain the total scan data volume corresponding to the target execution plan segment. The concurrent computing parameter acquisition module is used to obtain the number of available processor cores and the preset baseline data volume parameters; The concurrency decision calculation module is used to determine the target concurrency value of the target execution plan segment based on the numerical relationship between the total scan data volume, the baseline data volume parameter, and the number of available processor cores; The parallel scheduling module for execution plans is used to execute the target execution plan segment according to the target concurrency value.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a task concurrency dynamic adjustment program stored in the memory and executable on the processor. When the task concurrency dynamic adjustment program is executed by the processor, it implements the steps of the task concurrency dynamic adjustment method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a task concurrency dynamic adjustment program, which, when executed by the processor, implements the steps of the task concurrency dynamic adjustment method as described in any one of claims 1-7.

Images