Batch-mode-oriented data segmentation scheduling method and device, equipment and medium

Abstract

The embodiment of the invention discloses a batch-mode-oriented data segmentation scheduling method and device, equipment and a medium. A specific embodiment of the method comprises the steps of generating query execution plan information in response to query request information received by a query coordination node, and starting an asynchronous processing thread at the query coordination node; for the metadata information read in each iteration, executing the following steps: generating a data segmentation corresponding to the metadata information, and storing the corresponding data segmentation into a data segmentation cache structure; aiming at the segmentation request information, executing the following dynamic scheduling steps: dynamically selecting data segmentation from the data segmentation cache structure; sending the data segmentation distribution information to the corresponding execution nodes; and in response to meeting the preset condition, completing a data segmentation scheduling operation corresponding to the metadata information. According to the embodiment, on-demand generation and dynamic scheduling of data segmentation can be realized, memory overhead during batch mode data processing is saved, and scheduling waiting time is shortened.

Description

Data partitioning and scheduling methods, devices, equipment, and media for batch mode Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically to a data segmentation and scheduling method, apparatus, device, and medium for batch mode. Background Technology

[0002] Currently, in the field of distributed data processing and analysis, existing data partitioning and scheduling methods typically involve query coordination nodes centrally generating query plans and reading all metadata from external data sources at once. Based on this, all data partitions are centrally generated and then pre-statically allocated to each execution node. After receiving the allocation, each node begins to execute data reading and querying in parallel.

[0003] However, when using the above method for data partitioning and scheduling, the following technical problems often occur: the memory pressure on the query coordination node is high; the allocation cannot be dynamically adjusted according to the real-time load of the node, resulting in low resource utilization; the execution node has to wait for all partitions to be generated before it can start working, resulting in large startup delays and limited overall execution efficiency.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a data segmentation and scheduling method, apparatus, device, and medium for batch mode to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a data segmentation scheduling method for batch mode, including: in response to a query coordination node receiving query request information sent by a client, generating query execution plan information, and starting an asynchronous processing thread on the query coordination node to iteratively read metadata information of an external data source in batch mode; for each iteration of read metadata information, performing the following steps: generating data segments corresponding to the metadata information, and storing the corresponding data segments in a data segmentation cache structure, wherein the data segments stored in the data segmentation cache structure are not pre-bound to any execution node; in response to the query coordination node receiving segmentation request information sent by any of the multiple execution nodes, performing the following dynamic scheduling steps for the segmentation request information: dynamically selecting data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information; sending the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; and in response to satisfying preset conditions, completing the data segmentation scheduling operation corresponding to the metadata information.

[0008] Secondly, some embodiments of this disclosure provide a data segmentation and scheduling apparatus for batch mode, comprising: a generation and initiation unit configured to generate query execution plan information in response to a query coordination node receiving query request information sent by a client, and to initiate an asynchronous processing thread at the query coordination node to iteratively read metadata information of an external data source in batch mode; and an execution unit configured to perform the following steps for each iteration of read metadata information: generating data segments corresponding to the metadata information, and storing the corresponding data segments in a data segmentation cache structure, wherein the data segments stored in the data segmentation cache structure are not pre-bound to any execution node; in response to the query coordination node receiving segmentation request information sent by any of the multiple execution nodes, performing the following dynamic scheduling steps for the segmentation request information: dynamically selecting data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information; sending the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; and completing the data segmentation scheduling operation corresponding to the metadata information in response to satisfying preset conditions.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0011] The above embodiments of this disclosure have the following beneficial effects: The data segmentation and scheduling method for batch mode in some embodiments of this disclosure achieves on-demand generation and dynamic scheduling of data segments, saving memory overhead during batch mode data processing and reducing scheduling waiting time. Specifically, the reasons for high memory overhead and long scheduling waiting time during batch mode data processing are: traditional centralized generation methods require loading all data segments into the coordinating node's memory at once and performing static pre-allocation, resulting in excessively high peak memory usage; simultaneously, the execution node must wait for all segments to be generated before starting processing, causing unnecessary serial waiting delays. Based on this, the data segmentation and scheduling method for batch mode in some embodiments of this disclosure first, in response to the query coordinating node receiving query request information sent by the client, generates query execution plan information and starts an asynchronous processing thread on the query coordinating node to iteratively read the metadata information of the external data source in batch mode. By starting the asynchronous processing thread to iteratively read the metadata, on-demand and incremental generation of data segments is achieved. This avoids the memory peak pressure caused by the query coordinating node loading all metadata at once, significantly reducing memory usage and allowing the query execution process to start earlier. Then, for the metadata information read in each iteration, the following steps are performed: generating data segments corresponding to the aforementioned metadata information, and storing the corresponding data segments in a data segment cache structure. The data segments stored in the data segment cache structure are not pre-bound to any execution node. Generating corresponding data segments for each iteration and storing them in a cache structure not bound to execution nodes achieves the separation of task generation and resource allocation, making the data segments a dynamically schedulable shared resource pool, laying the foundation for subsequent load balancing scheduling. Secondly, in response to the query coordination node receiving a segmentation request from any of the multiple execution nodes, the following dynamic scheduling steps are performed: based on the real-time load information of the execution node corresponding to the segmentation request information, dynamically selecting data segments from the aforementioned data segment cache structure to generate data segmentation allocation information. Dynamically selecting data segments from the cache based on the real-time load information of the execution nodes. This allows scheduling to adapt to the current processing capacity of nodes, allocating more tasks to idle or high-efficiency nodes, thereby achieving dynamic load balancing and improving overall resource utilization. Next, the aforementioned data partitioning and allocation information is sent to the corresponding execution nodes to perform data reading and query operations. Sending the allocation information to the corresponding execution nodes to immediately execute data reading and querying achieves a tight connection between task allocation and execution. Furthermore, the parallel processing capabilities of the nodes effectively shorten query response time and improve overall processing throughput. Finally, in response to the fulfillment of preset conditions, the data partitioning and scheduling operation corresponding to the aforementioned metadata information is completed.The scheduling ends when the preset completion conditions are met (e.g., all data has been split and processed), ensuring the integrity and correctness of the query task and forming a closed loop. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of the batch-oriented data segmentation and scheduling method according to the present disclosure; Figure 2 is a structural schematic diagram of some embodiments of the batch-oriented data segmentation and scheduling apparatus according to the present disclosure; Figure 3 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Referring to Figure 1, a flowchart 100 of some embodiments of the batch-mode oriented data partitioning and scheduling method according to this disclosure is shown. This batch-mode oriented data partitioning and scheduling method includes the following steps: Step 101, in response to the query coordinating node receiving query request information sent by the client, generating query execution plan information, and starting an asynchronous processing thread on the query coordinating node to iteratively read metadata information of the batch mode from the external data source; In some embodiments, the execution entity (e.g., an electronic device) of the above-described batch-mode oriented data partitioning and scheduling method can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitations are made here.

[0021] In other embodiments, the execution entity may, in response to the query coordinating node receiving query request information from the client, generate query execution plan information and start an asynchronous processing thread on the query coordinating node to iteratively read metadata information from the external data source in batch mode. The query coordinating node may be a central scheduling unit that receives and manages queries. For example, the query coordinating node may be the Spark SQL Driver. The client may be a user or application (e.g., Jupyter Notebook) that initiates the query request. The query request information may be request data including query statements and parameters. For example, the query request information may be an SQL query in JSON format. The query execution plan information may be a detailed scheme guiding how the query is executed. The asynchronous processing thread may be a background worker thread independent of the main program flow, capable of executing specific tasks in parallel without blocking the main thread. For example, the asynchronous processing thread may be a thread specifically designed to scan HDFS metadata. The external data source may be a system or service storing the raw data to be processed. For example, the external data source may be an HDFS file system, an Amazon S3 bucket, or a MySQL database. The metadata information may be descriptive information about the batch data in the external data source, and may include: a file list, partition structure, and size. For example, the metadata information mentioned above could be a list of paths and sizes of files in each partition of a Hive table.

[0022] In addressing the technical problems mentioned above by adopting technical solutions, the application scenario—interactive analysis and querying of large-scale offline data warehouses—often presents the following technical challenges: traditional queries generate static, generic query execution plans, failing to fully consider the locality of data distribution and the real-time status of cluster resources in batch mode, leading to insufficient parallelism and low resource utilization, thus increasing response time. To meet the specific requirements of this application scenario—high throughput, low latency, and high resource utilization—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can, in response to the query coordination node receiving query request information from the client, generate query execution plan information and initiate an asynchronous processing thread on the query coordination node to iteratively read metadata information from the external data source in batch mode. This can include the following steps: First, in response to the query coordination node receiving query request information from the client, generate query semantic information for the query request information. This query semantic information can be a structured representation of the query intent. For example, the query semantic information can be the parsed SELECT field, FROM table, and WHERE condition. In practice, first, the query coordinating node parses the raw request sent by the client (e.g., an SQL string). Then, it extracts key operation elements (e.g., SELECT, FROM, WHERE). Finally, it generates structured query semantic information (e.g., an Abstract Syntax Tree (AST)).

[0023] The second step is to determine the target external data source based on the query semantic information described above. This target external data source can be the specific data storage location that the query request seeks to access, and can be specified by a unique resource identifier (URI). In practice, first, the data source to be accessed is determined based on the table name or path in the query semantic information, serving as the target external data source (e.g., a specific Hive table or HDFS path).

[0024] The third step is to obtain the metadata structure information of the target external data source. This metadata structure information can be a description of the target external data source's organizational structure, and may include: table field names, types, and partition keys. In practice, first, connect to the target external data source (e.g., Hive Metastore). Then, obtain the detailed metadata structure of the target external data source (e.g., schema, partition list, file format).

[0025] The fourth step involves generating a query execution logic diagram based on the aforementioned query semantic information and metadata structure information. This query execution logic diagram can be a logical flowchart describing the order of query execution, composed of abstract operators (e.g., scan, filter, join). For example, it could be a Directed Acyclic Graph (DAG) composed of Scan, Filter, and Project operators. In practice, first, the query semantic information (what needs to be done) is combined with the metadata structure information (what the data looks like). Then, a logical operation diagram describing the entire data processing flow is constructed (e.g., a DAG including operators such as scan, filter, aggregation, and join). Finally, the query execution logic diagram is generated.

[0026] The fifth step involves generating a parallelizable optimized scan subtree from the query execution logic graph based on the batch mode configuration parameters. These batch mode configuration parameters can be configurable parameters used to control and optimize batch data processing behavior. For example, they could be the amount of data processed each time or the number of parallel tasks. The parallelizable optimized scan subtree can be a portion of the query execution logic graph that is identified as suitable for being split into multiple parallel tasks for simultaneous execution. In practice, first, the pre-configured batch mode parameters (e.g., max_split_size) are read. Then, each data scan operation node is identified in the query execution logic graph. Finally, each node is marked and reorganized into an optimized scan subtree that can be executed independently in parallel.

[0027] Step 6: Based on the data distribution statistics from the external data source, perform data locality optimization on the parallelizable optimizable scan subtree to generate a partition-aware execution logic graph. The data distribution statistics can be statistics on the physical distribution of data in the storage system, such as whether the data is partitioned, the size of each partition, and the locality of data blocks. The partition-aware execution logic graph can be an optimized graph incorporating data location information. In practice, first, obtain the latest data distribution statistics from the external data source (e.g., through the NameNode or a statistics service). Then, reconstruct the parallelizable optimizable scan subtree based on the data distribution statistics (e.g., data block locations). Finally, generate a partition-aware execution logic graph that clearly knows where the data is and how it is grouped for scanning.

[0028] Step 7: Based on the partition-aware execution logic graph, generate a physical execution plan to identify the aforementioned physical execution plan as the query execution plan information. This physical execution plan can be a final execution scheme that transforms the logical execution plan into a runnable implementation on a specific computing engine, including specific implementation details (e.g., algorithm, data format, resource allocation). For example, the physical execution plan can explicitly specify: using Parquet format to read data, employing the Sort-Merge algorithm for table joins, and allocating specific memory and CPU resources for each task.

[0029] Step 8: Based on the query execution plan described above, start the asynchronous processing thread. In practice, first, after the physical execution plan is generated, the query coordinating node creates a separate background thread. Then, the physical execution plan is passed as input to this thread. Finally, the asynchronous processing thread is started.

[0030] The ninth step involves iteratively reading the aforementioned metadata information using the asynchronous processing thread. First, the asynchronous processing thread locates the external data source to be scanned based on the physical execution plan. Then, it iteratively acquires metadata information (e.g., a list of file paths) step by step (e.g., reading one partition or a batch of files at a time). Finally, the metadata acquired in each iteration is output for subsequent data segmentation.

[0031] The above-described steps, as an inventive point of this disclosure, solve the technical problem that "traditional queries generate static, generic query execution plans, failing to fully consider the locality of data distribution and the real-time status of cluster resources in batch mode, resulting in insufficient parallelism and low resource utilization, and increased response time." The reasons for this technical problem are as follows: the traditional plan generation process is disconnected from data distribution and system resources, and the plan lacks flexibility. This invention, by deeply integrating data distribution characteristics and batch processing requirements in the plan generation stage, achieves the generation of physical execution plans with data locality awareness and high parallelism potential, saving computational resources and time overhead caused by remote data transmission and insufficient parallelism.

[0032] Step 102: For the metadata information read in each iteration, perform the following steps: Step 1021: Generate the data segmentation corresponding to the metadata information, and store the corresponding data segmentation in the data segmentation cache structure.

[0033] In some embodiments, the execution entity can generate data partitions corresponding to the metadata information and store these data partitions in a data partition cache structure. The data partitions stored in the data partition cache structure are not pre-bound to any execution node. The data partitioning can be the division of large data blocks (metadata information) into logical task units that can be processed independently and in parallel. For example, the data partitioning can be a 1TB Hive table divided into 100 independent data scanning tasks. The data partition cache structure can be a shared memory storage area for temporarily storing generated but unallocated data partitions, such as a thread-safe queue or hash table. The execution node can be a worker node that actually performs data reading and computation tasks, responsible for processing the allocated data partitions. For example, the execution node can be an Executor process in a Spark cluster or a TaskManager in Flink.

[0034] In some optional implementations of certain embodiments, the aforementioned execution entity can generate data segments corresponding to the aforementioned metadata information and store the corresponding data segments in a data segmentation cache structure. This can include the following steps: First, based on the aforementioned metadata information, determine the corresponding storage location information. The aforementioned storage location information can be information describing the physical location of the metadata information in the underlying storage system, and may include: protocol, path, endpoint. In practice, firstly, the location identifier in the metadata information (e.g., database table name, file path) is determined. Then, it is converted into a standardized storage location format according to the data source type to obtain the storage location information.

[0035] The second step, based on the aforementioned storage location information, is to determine the corresponding data access range information. This data access range information can be a precise description defining which portion of data to read from the storage location, and may include: offset, length, and partition key value. In practice, first, the storage system is connected to obtain a list of data blocks or file sizes. Then, a reasonable access range is determined based on the batch mode configuration (e.g., a maximum of 128MB per partition). Finally, a precise description is generated for each range to obtain the data access range information.

[0036] The third step is to generate a data segmentation identifier based on the aforementioned data range information. This identifier can be a unique string or numeric ID that identifies the data segment. In practice, firstly, key information is combined using a consistent naming convention (e.g., storage location + range), and then a globally unique identifier is generated as the data segmentation identifier.

[0037] The fourth step is to generate data segmentation metadata based on the aforementioned storage location information and data access scope information. This data segmentation metadata can be structured metadata describing data segmentation attributes, including format, size, and estimated row count. In practice, first, storage location information, data access scope information, and data format can be obtained. Then, these are organized into structured objects. Finally, the construction of the data segmentation metadata is completed.

[0038] The fifth step is to generate data segments based on the aforementioned data segmentation identifiers and metadata. In practice, first, a data segmentation object is instantiated. Then, the data segmentation identifiers and metadata are populated as attributes, and finally, the data segments are generated.

[0039] The sixth step involves compressing and encoding the data segments to generate a serialized data segment object. This serialized data segment object can be converted into a byte stream or a more compact in-memory representation for network transmission or persistent storage. In practice, Protocol Buffers or Avro are used to encode the metadata data segment object into a byte array.

[0040] Step 7: Determine the hash signature of the serialized data segment object to generate a unique fingerprint for the data segment. The hash signature can be a fixed-length digest value obtained by applying a hash algorithm to the serialized data segment object, used for integrity verification and deduplication. The unique fingerprint can be a globally unique identifier composed of the hash signature and other identifying information, used for fast comparison and data deduplication. In practice, first, a hash algorithm (e.g., SHA-256) is applied to the serialized byte array to determine a fixed-length hash value. Then, the hash value is converted into a readable string format to serve as the unique fingerprint for the data segment. For example, the SHA-256 value a1b2c3d4e5f6... is obtained and used as the unique fingerprint for the data segment.

[0041] Step 8: Associate the unique fingerprint of the data segment with the serialized data segment object and store them in the data segment cache structure. In practice, first, a key-value pair is constructed using the unique fingerprint of the data segment as the key and the serialized data segment object as the value. Then, this key-value pair is inserted into a thread-safe cache structure. Finally, the relevant indexes are updated.

[0042] In some optional implementations of certain embodiments, the execution entity can generate data segmentation metadata based on the storage location information and the data access range information, which may include the following steps: First, generate a basic data segmentation description framework based on the storage location information and the data access range information. The basic data segmentation description framework can be a standardized structural template for data segmentation, which may include: location, range, and format. In practice, first, the data source protocol and path are obtained by parsing the storage location information. Then, the segmentation boundaries are determined based on the data access range. Finally, standardized fields are filled to generate the description framework, which serves as the basic data segmentation description framework.

[0043] The second step involves generating data read performance parameters based on the preset data source type configuration and the aforementioned data access range information. These performance parameters can be estimated resource consumption metrics for reading the data segment; for example, they could be I / O latency or network transmission volume. In practice, first, the data source type configuration is determined (e.g., network latency needs to be considered for S3). Then, the estimated read time is determined based on the access range size. Finally, the data read performance parameters are generated.

[0044] The third step involves generating data segmentation metadata based on pre-acquired historical execution statistics and the aforementioned data read performance parameters. The historical execution statistics can be performance data collected from execution records of similar data segments in the past. In practice, first, execution records with similar data segments to the historical execution statistics are queried. Then, the current performance parameters are compared and calibrated with the historical data. Finally, optimized metadata is generated. For example, if the historical average read time for a 128MB file in the same S3 region is 2.0s, the current estimated time of 1.8s is calibrated to 2.0s.

[0045] Step 1022: In response to the query coordination node receiving a splitting request information sent by any of the multiple execution nodes, the following dynamic scheduling steps are performed for the splitting request information: Step 10221: Based on the real-time load information of the execution node corresponding to the splitting request information, data splitting is dynamically selected from the data splitting cache structure to generate data splitting allocation information.

[0046] In some embodiments, the execution entity can dynamically select data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information. The segmentation request information can be a message actively sent by the execution node requesting the allocation of data segments. For example, the segmentation request information can be an RPC request including the node ID and the current queue length. The real-time load information can be a measure of the execution node's real-time resource usage at the time the request is received, such as CPU utilization or memory pressure. The data segmentation allocation information can be an instruction generated by a query coordinating node, indicating that a specific data segment should be allocated to a specific execution node.

[0047] In addressing the technical problems mentioned above by adopting technical solutions, the application scenario—a cloud-native data analytics platform with high concurrency and drastic load fluctuations—often presents the following technical issues: static task allocation strategies cannot adapt to the rapid dynamic changes in cluster node resources, leading to overload on some nodes while others remain idle, resulting in decreased overall throughput and increased task execution latency. Considering the following requirements for this application scenario—high elasticity, high resource utilization, and stable service quality—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can dynamically select data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information. This can include the following steps: First, based on the segmentation request information, determine the corresponding execution node identifier. The execution node identifier can be a symbol that uniquely identifies an execution node, used for addressing and status tracking. In practice, firstly, the coordinating node receives and parses the segmentation request information sent by the execution node. Then, the sender's identity is extracted from the message header or specific fields of the segmentation request information. Finally, the identity identifier is confirmed as the execution node identifier corresponding to this request.

[0048] The second step is to determine the real-time load information of the corresponding execution node based on the aforementioned execution node identifier. In practice, firstly, the cluster resource manager or the monitoring agent of the corresponding execution node itself is queried based on the execution node identifier. Then, a snapshot of the key performance indicators of the corresponding execution node at the current moment is obtained to obtain the real-time load information.

[0049] The third step involves generating a threshold for the number of data splits that can be allocated, based on the aforementioned real-time load information. This threshold can be the maximum number of data splits that can be dynamically calculated based on the current load of the execution node, representing the maximum number of data splits that can be allocated to it in this instance. For example, based on a high CPU utilization of 85%, it can be determined that a maximum of two data splits can be allocated to Worker-3 in this instance. In practice, firstly, the system reads the preset load-allocation mapping strategy. Then, based on real-time load information (e.g., CPU utilization), it calculates the upper limit of the number of tasks suitable for allocation to this node. Finally, this upper limit is determined as the data splitting threshold.

[0050] The fourth step involves filtering candidate data segments from the data segmentation cache structure based on the aforementioned data segmentation quantity threshold to generate a candidate data segmentation set. This candidate data segmentation set can be a preliminary selection of data segments from the data segmentation cache structure that meet the current allocation conditions. In practice, firstly, all currently available data segments are retrieved from the data segmentation cache structure. Then, based on the data segmentation quantity threshold, a corresponding number of segments are randomly selected or in FIFO order. Finally, the selected segments are grouped into a candidate data segmentation set.

[0051] The fifth step involves prioritizing the candidate data splits based on the preset local policy information and the execution node identifiers, generating a priority ranking result. The local policy information can be a set of rules guiding task priority allocation; for example, it can prioritize splits where data and execution nodes reside on the same physical rack. The priority ranking result can be an ordered list of candidate data splits, arranged from highest to lowest priority, obtained after applying the local policy. For example, the priority ranking result could be sorted as: Split-A (same rack), Split-C (same rack), Split-B (across racks). In practice, for example, first, preset local policy information (e.g., data locality priority) is obtained. Then, combined with the execution node identifiers (used to determine node locations), the priority score of each data split in the candidate set relative to that node is calculated. Finally, the candidate set is sorted in descending order based on the scores to generate the priority ranking result.

[0052] Step 6: Based on the priority ranking results described above, select the target data segment from the candidate data segmentation set to generate the target data segment. The target data segment can be the specific data segment that has been sorted and selected, and ultimately assigned to the execution node. In practice, first, obtain the priority ranking results. Then, directly select one or more data segments from the top of the ranking list (i.e., the highest priority position). Finally, determine these selected segments as the target data segment.

[0053] The seventh step involves encapsulating the target data segment and the execution node identifier to generate data segmentation and allocation information. In practice, firstly, the target data segment is associated with the execution node identifier determined in the first step. Then, it is encapsulated into a structured allocation instruction message. Finally, the allocation instruction message is used as the generated data segmentation and allocation information.

[0054] The above-described steps, as an inventive point of this disclosure, solve the technical problem mentioned in the background art: "Static task allocation strategies cannot adapt to the rapid dynamic changes in cluster node resources, leading to some nodes being overloaded while others are idle, resulting in a decrease in overall throughput and an increase in task execution latency." The reasons for this technical problem are as follows: task allocation decisions are disconnected from the real-time operating status of the execution nodes, and the allocation strategy lacks flexibility. This invention, by quantifying the real-time load and using it as the core input to dynamically adjust the task allocation amount each time, achieves refined and flexible task allocation based on the real-time processing capabilities of nodes, saving on idle computing resources and task queuing time caused by uneven node load.

[0055] In addressing the technical challenges of the aforementioned background technologies, and considering the application scenario—a cloud data warehouse with mixed workloads, such as an e-commerce company using Snowflake to handle daily real-time order analysis, user behavior log queries, and month-end financial statement generation—the following technical issues often arise: simple task allocation strategies struggle to balance data locality (order tables are partitioned and stored in different regions), historical execution efficiency, and current system load (sudden surges in queries during promotions), leading to prolonged query response times and wasted idle resources. To meet the specific requirements of this application scenario—high throughput and stable execution performance—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can filter candidate data segments from the data segmentation cache structure based on the aforementioned data segmentation quantity threshold to generate a candidate data segmentation set. This can include the following steps: First, based on the aforementioned execution node identifier and preset node-data affinity rules, an affinity data segmentation set is initially screened from the data segmentation cache structure to obtain the affinity data segmentation set. The aforementioned node-data affinity rules can be policy rules that prioritize the allocation of specific data partitions to specific nodes. For example, these rules could prioritize the allocation of data stored on a data node in rack A to compute nodes within the same rack. The aforementioned affinity data partition set can be a set of data partitions with a high "match" to the execution node, initially selected based on the node-data affinity rules. In practice, first, the coordinating node is queried based on the execution node identifier (e.g., hostname host-10) and preset node-data affinity rules (e.g., "prioritize allocation of data within the same rack"). Then, the data partition cache structure is traversed, and the storage location metadata of each data partition is checked. Finally, the data partition IDs that conform to the node-data affinity rules are collected to form an affinity data partition set.

[0056] The second step involves using the historical execution performance database to determine the corresponding historical execution time statistics, based on the aforementioned execution node identifiers and the characteristics of each data segment in the affinity data segmentation set. The characteristics of the data segment can be information describing the attributes of the data segment itself. For example, the characteristics of the data segment could be: data format (Parquet), compression method (Snappy), data size (128MB), and the data source (table sales_fact). The historical execution performance database can be a database storing all past data segment execution records (e.g., execution time, resource usage). For example, the historical execution performance database could include which node each data segment was executed on and how long it took in the past month. The above can be task time metrics analyzed from the historical database. For example, the average execution time for a certain type of data segment (e.g., Parquet format, 128MB size) in the past 100 executions was 2.1 seconds. In practice, firstly, for each data segment in the affinity data segmentation set, its key characteristics (e.g., data source, format, size) are extracted. Then, the historical execution performance database is queried using these key characteristics as query conditions. Finally, obtain the historical execution time statistics for this feature combination.

[0057] The third step involves updating the dynamic time threshold for the current scheduling period based on the current average system load and the historical execution time statistics mentioned above. The current average system load can be the average resource utilization of the entire cluster within the most recent time window. For example, the current average system load could be the average CPU utilization of all nodes in the cluster over the past 5 minutes, which is 65%. The current scheduling period can be the time window or batch corresponding to this scheduling decision. The dynamic time threshold can be the upper limit of task execution time dynamically calculated based on the current system state. The updated dynamic time threshold can be the final time threshold adjusted based on historical execution time statistics and the current average system load. In practice, first, the average CPU and memory utilization of each node is obtained to calculate the average system load. Then, based on this load level, the baseline threshold calculated based on historical execution time statistics is dynamically adjusted (the threshold is tightened when the load is high). Finally, the updated dynamic time threshold applicable to the current scheduling period is obtained. For example, if the baseline threshold is 2 seconds and the current system load is 70% (relatively high), the threshold is tightened by 20% according to predefined rules, resulting in an updated threshold of 1.6 seconds.

[0058] The fourth step involves filtering inefficient data segments from the affinity data segment set whose estimated execution time exceeds the updated dynamic time threshold, based on the updated dynamic time threshold. This filters the data segments to generate a filtered affinity data segment set. The estimated execution time can be the predicted execution time required for a data segment under current conditions. For example, the estimated execution time could be 2.3 seconds based on historical statistics (2.1 seconds) and the current load factor (1.1 times). Inefficient data segments are those whose estimated execution time exceeds the dynamic time threshold and may become performance bottlenecks. The filtered affinity data segment set is the set of data segments remaining after removing inefficient segments from the affinity set. In practice, first, the estimated execution time of each data segment in the affinity data segment set is compared with the updated dynamic time threshold. Then, all data segments with estimated times exceeding the threshold are filtered out. Finally, the filtered affinity data segment set is generated.

[0059] Fifth, in response to the fact that the filtered affinity segment set does not reach the threshold for the number of data segments, supplementary data segments are selected from the general ready queue of the data segmentation cache structure to obtain a supplementary data segment set. The general ready queue can be a queue of "ready" but not categorized according to affinity rules in the data segmentation cache structure. The supplementary data segment set can be a result set of data segments selected from the general ready queue and added to the filtered affinity set to reach the threshold for the number of data segments. In practice, first, the size of the filtered affinity segment set is checked. If it is less than the threshold for the number of data segments (e.g., 10 are needed, but only 7 are available), the difference is calculated. Then, the difference in number of data segments is selected sequentially from the head of the general ready queue (which stores all unassigned segments) maintained by the data segmentation cache structure. Finally, these are merged with the filtered affinity set to form the supplementary data segment set. For example, take 3 segments from the general queue and merge them with the original 7 affinity segments to get a set containing 10 segments.

[0060] The sixth step involves performing deduplication and conflict checks on the supplemented data partition set to generate a candidate data partition set. In practice, firstly, all data partition IDs in the supplemented data partition set are checked to remove duplicates. Then, a comparison is made with a global temporary lock record that is either "being allocated" or "allocated" to perform conflict checks and remove partitions that have been locked by other scheduling threads. Finally, a candidate data partition set is generated.

[0061] The above-described steps, as an inventive point of this disclosure, solve the aforementioned technical problem: "Simple task allocation strategies struggle to achieve a balance among multiple dimensions such as data locality, historical execution efficiency, and current system load, leading to prolonged query response time and wasted idle resources." The reasons for this technical problem are as follows: the task selection criteria are singular and static, unable to adapt to multi-dimensional and dynamically changing optimization goals. This invention, by introducing a multi-level, adaptive filtering and supplementation mechanism, achieves the generation of a candidate set that intelligently avoids potential slow tasks and ensures full resource utilization while maintaining data locality. This saves tail delay time caused by executing slow tasks and resource waste caused by insufficient task allocation.

[0062] Step 10222: Send the data splitting and allocation information to the corresponding execution node to perform data reading and query operations.

[0063] In some embodiments, the aforementioned execution entity may send the data partitioning and allocation information to the corresponding execution node to perform data reading and query operations. These data reading and query operations may involve the execution node reading data from the storage system based on the allocation information and performing calculations on the corresponding operators in the query plan.

[0064] In some optional implementations of certain embodiments, the aforementioned execution entity can send the aforementioned data partitioning and allocation information to the corresponding execution node to perform data reading and query operations. This may include the following steps: First, generate a target execution node identifier based on the aforementioned data partitioning and allocation information. The target execution node identifier may be a unique identifier of the execution node responsible for processing the data partitioning, as specified in the data partitioning and allocation information. In practice, first, the data structure of the data partitioning and allocation information is parsed. Then, node identification information is extracted from a specific field (e.g., assigned_node), formatted into a standard identifier used internally by the system, and used as the target execution node identifier.

[0065] The second step is to determine the available execution node communication endpoints based on the target execution node identifier. These execution node communication endpoints can be network address information used to establish network connections with the execution node, such as IP addresses and ports. In practice, first, the target execution node identifier is used to query the cluster's node registry or service discovery component. Then, the list of currently reported, available network addresses of that node is obtained to determine the execution node communication endpoints.

[0066] The third step is to establish a bidirectional communication channel based on the aforementioned execution node communication endpoint. This bidirectional communication channel can be a persistent network connection between the coordinating node and the execution node that supports bidirectional data transmission. In practice, firstly, the execution node communication endpoint is used as the target address. Then, a network connection request is initiated, and a bidirectional communication channel supporting asynchronous bidirectional communication is negotiated and established.

[0067] The fourth step involves asynchronously sending the data splitting and allocation information to the corresponding execution node via the aforementioned bidirectional communication channel, and asynchronously receiving the allocation confirmation information returned by the execution node via the same bidirectional communication channel. This allocation confirmation information can be a confirmation receipt returned by the execution node to the coordinating node after successfully receiving and parsing the data splitting and allocation information. In practice, firstly, the data splitting and allocation information is sent to the execution node asynchronously (without blocking the main thread) via the bidirectional communication channel. Then, the allocation confirmation information returned by the execution node is listened for and received asynchronously on the same channel. Finally, the allocation confirmation information is obtained.

[0068] Fifth, in response to the aforementioned allocation confirmation information received by the coordination node, the data segment identifier to be updated and the target status flag are extracted based on the allocation confirmation information. The data segment identifier to be updated can be the ID of the data segment whose status needs to be updated, included in the allocation confirmation information. The target status flag can be the final status identifier that the data segment needs to be updated to in the coordination node's cache record after it has been successfully allocated to the execution node. In practice, first, the allocation confirmation information is parsed. Then, the data segment identifier field to be updated and the target status flag indicating "allocated" or "confirmed" are extracted from the allocation confirmation information.

[0069] Step 6: Based on the data segmentation identifier to be updated, lock the corresponding data segmentation record in the data segmentation cache structure. This data segmentation record can be an information unit in the data segmentation cache structure representing a data segmentation entry and its current state (e.g., pending allocation, allocated). In practice, first, based on the data segmentation identifier, find the corresponding record in the data segmentation cache structure and use it as the data segmentation record. Then, apply an exclusive lock to the data segmentation record to prevent other threads from concurrently modifying it during the state update process. Finally, the data segmentation record is successfully locked.

[0070] Step 7: Atomically update the state of the data segmentation record to the target state flag, and release the data segmentation record. In practice, first, while the data segmentation record is locked, atomically update its current state field to the target state flag. Then, perform necessary association operations (e.g., log recording). Finally, release the lock on the record.

[0071] Step 8: Send data operation execution commands to the corresponding execution nodes to perform data read and query operations. These data operation execution commands can be trigger instructions issued by the coordinating node to the execution nodes, instructing them to begin data read and query calculations. In practice, first, confirm that the data partitioning status has been successfully updated. Then, send a formal "start execution" command to the execution nodes through a bidirectional communication channel. Finally, upon receiving the command, the execution nodes initiate the data read and query operations.

[0072] Step 10223: In response to the fulfillment of preset conditions, complete the data splitting and scheduling operation corresponding to the metadata information.

[0073] In some embodiments, the execution entity may complete the data partitioning and scheduling operation corresponding to the metadata information in response to the fulfillment of preset conditions. These preset conditions may be pre-defined rules or states used to determine whether the data partitioning and scheduling can be terminated. For example, the preset conditions may be that all data partitions have been processed and the data partitioning cache structure is empty. The data partitioning and scheduling operation may refer to the entire process from generating data partitions to allocating and executing them. For example, the data partitioning and scheduling operation may be a large-scale aggregation query involving 1000 data partitions, from the start of scheduling to the completion of each task.

[0074] As an example: First, the query coordinating node continuously monitors whether preset conditions are met (whether the asynchronous processing thread has completed reading all metadata and whether the data splitting cache structure is empty). Then, if and only if both conditions are met, the coordinating node determines that the data splitting scheduling operation is complete and triggers the subsequent query result aggregation process.

[0075] In some optional implementations of certain embodiments, the aforementioned execution entity may further perform the following steps: First, in response to the failure to meet preset conditions, it receives the next splitting request information sent by the execution node, and confirms the next splitting request information as a splitting request, continuing to execute the aforementioned dynamic scheduling steps. The next splitting request information may be a request actively sent by the execution node after completing the processing of the current batch of data to obtain a new task. In practice, firstly, the query coordinating node continuously listens for execution node requests when the preset conditions are not met. When it receives the next splitting request information sent by an execution node (e.g., executor-5), the query coordinating node confirms this request information as the splitting request information that needs to be processed. Then, the query coordinating node re-executes the dynamic scheduling steps with the latest request (the next splitting request information) as input. Finally, it sends the new data splitting and allocation information to the corresponding execution node to begin a new round of processing.

[0076] The above embodiments of this disclosure have the following beneficial effects: The data segmentation and scheduling method for batch mode in some embodiments of this disclosure achieves on-demand generation and dynamic scheduling of data segments, saving memory overhead during batch mode data processing and reducing scheduling waiting time. Specifically, the reasons for high memory overhead and long scheduling waiting time during batch mode data processing are: traditional centralized generation methods require loading all data segments into the coordinating node's memory at once and performing static pre-allocation, resulting in excessively high peak memory usage; simultaneously, the execution node must wait for all segments to be generated before starting processing, causing unnecessary serial waiting delays. Based on this, the data segmentation and scheduling method for batch mode in some embodiments of this disclosure first, in response to the query coordinating node receiving query request information sent by the client, generates query execution plan information and starts an asynchronous processing thread on the query coordinating node to iteratively read the metadata information of the external data source in batch mode. By starting the asynchronous processing thread to iteratively read the metadata, on-demand and incremental generation of data segments is achieved. This avoids the memory peak pressure caused by the query coordinating node loading all metadata at once, significantly reducing memory usage and allowing the query execution process to start earlier. Then, for the metadata information read in each iteration, the following steps are performed: generating data segments corresponding to the aforementioned metadata information, and storing the corresponding data segments in a data segment cache structure. The data segments stored in the data segment cache structure are not pre-bound to any execution node. Generating corresponding data segments for each iteration and storing them in a cache structure not bound to execution nodes achieves the separation of task generation and resource allocation, making the data segments a dynamically schedulable shared resource pool, laying the foundation for subsequent load balancing scheduling. Secondly, in response to the query coordination node receiving a segmentation request from any of the multiple execution nodes, the following dynamic scheduling steps are performed: based on the real-time load information of the execution node corresponding to the segmentation request information, dynamically selecting data segments from the aforementioned data segment cache structure to generate data segmentation allocation information. Dynamically selecting data segments from the cache based on the real-time load information of the execution nodes. This allows scheduling to adapt to the current processing capacity of nodes, allocating more tasks to idle or high-efficiency nodes, thereby achieving dynamic load balancing and improving overall resource utilization. Next, the aforementioned data partitioning and allocation information is sent to the corresponding execution nodes to perform data reading and query operations. Sending the allocation information to the corresponding execution nodes to immediately execute data reading and querying achieves a tight connection between task allocation and execution. Furthermore, the parallel processing capabilities of the nodes effectively shorten query response time and improve overall processing throughput. Finally, in response to the fulfillment of preset conditions, the data partitioning and scheduling operation corresponding to the aforementioned metadata information is completed.The scheduling ends when the preset completion conditions are met (e.g., all data has been split and processed), ensuring the integrity and correctness of the query task and forming a closed loop.

[0077] Referring further to Figure 2, as an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a batch-mode data segmentation and scheduling device, which correspond to the method embodiments shown in Figure 1. This batch-mode data segmentation and scheduling device can be specifically applied to various electronic devices.

[0078] As shown in Figure 2, a data segmentation and scheduling device 200 for batch mode includes a generation and startup unit 201 and an execution unit 202. The generation and startup unit 201 is configured to: generate query execution plan information in response to a query coordination node receiving a query request from a client, and start an asynchronous processing thread at the query coordination node to iteratively read metadata information from the external data source in batch mode. Execution unit 202 is configured to perform the following steps for the metadata information read in each iteration: generate data segments corresponding to the metadata information, and store the corresponding data segments in a data segmentation cache structure, wherein the data segments stored in the data segmentation cache structure are not pre-bound to any execution node; in response to the query coordination node receiving a segmentation request from any of the multiple execution nodes, perform the following dynamic scheduling steps for the segmentation request: dynamically select data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request to generate data segmentation allocation information; send the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; and complete the data segmentation scheduling operation corresponding to the metadata information in response to the satisfaction of preset conditions.

[0079] It is understood that the units described in the batch-mode data segmentation and scheduling apparatus 200 correspond to the steps in the method described with reference to FIG1. ​​Therefore, the operations, features, and beneficial effects described above for the method also apply to the image segmentation apparatus 200 and the units contained therein, and will not be repeated here.

[0080] Referring now to FIG3, a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure is shown. The electronic device shown in FIG3 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

[0081] As shown in Figure 3, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0082] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 shows electronic device 300 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Each box shown in Figure 3 may represent one device, or multiple devices may be represented as needed.

[0083] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0084] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0085] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0086] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following steps to be performed: In response to the query coordination node receiving query request information sent by the client, the electronic device generates query execution plan information and starts an asynchronous processing thread at the query coordination node to iteratively read metadata information from the external data source in batch mode; For each iteration of read metadata information, the electronic device performs the following steps: generates data segments corresponding to the aforementioned metadata information and stores the corresponding data segments in a data segmentation cache structure, wherein the data segments stored in the data segmentation cache structure are not pre-bound to any execution node; In response to the query coordination node receiving segmentation request information sent by any of the multiple execution nodes, the electronic device performs the following dynamic scheduling steps for the segmentation request information: Based on the real-time load information of the execution node corresponding to the segmentation request information, the electronic device dynamically selects data segments from the data segmentation cache structure to generate data segmentation allocation information; The electronic device sends the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; In response to the satisfaction of preset conditions, the electronic device completes the data segmentation scheduling operation corresponding to the aforementioned metadata information.

[0087] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0089] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a generation and startup unit 201 and an execution unit 202. The names of these units do not necessarily limit the specific unit itself. For instance, the generation and startup unit may also be described as "a unit that, in response to a query coordination node receiving query request information from a client, generates query execution plan information and starts an asynchronous processing thread on the aforementioned query coordination node to iteratively read metadata information from an external data source in batch mode."

[0090] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0091] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data partitioning and scheduling method for batch processing, comprising: In response to receiving a query request from a client, the query coordination node generates a query execution plan and starts an asynchronous processing thread on the query coordination node to iteratively read metadata information from the external data source in batch mode. For the metadata information read in each iteration, the following steps are performed: generating data segments corresponding to the metadata information, and storing the corresponding data segments in a data segmentation cache structure, wherein the data segments stored in the data segmentation cache structure are not pre-bound to any execution node; in response to the query coordination node receiving segmentation request information sent by any of the multiple execution nodes, the following dynamic scheduling steps are performed for the segmentation request information: dynamically selecting data segments from the data segmentation cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information; sending the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; and in response to the satisfaction of preset conditions, completing the data segmentation scheduling operation corresponding to the metadata information.

2. The method according to claim 1, wherein, The method further includes: in response to the failure to meet the preset conditions, receiving the next splitting request information sent by the execution node, and determining the next splitting request information as splitting request information, and continuing to execute the dynamic scheduling steps.

3. The method according to claim 1, wherein, The process of generating the data segment corresponding to the metadata information and storing the corresponding data segment in a data segmentation cache structure includes: determining the corresponding storage location information based on the metadata information; determining the corresponding data access range information based on the storage location information; generating a data segmentation identifier based on the data range information; generating data segmentation metadata based on the storage location information and the data access range information; generating a data segment based on the data segmentation identifier and the data segmentation metadata; compressing and encoding the data segment to generate a serialized data segment object; determining the hash signature of the serialized data segment object to generate a unique fingerprint of the data segment; and associating the unique fingerprint of the data segment with the serialized data segment object and storing it in the data segmentation cache structure.

4. The method according to claim 3, wherein, The step of generating data segmentation metadata based on the storage location information and the data access range information includes: generating a basic data segmentation description framework based on the storage location information and the data access range information; generating data read performance parameters based on a preset data source type configuration and the data access range information; and generating data segmentation metadata based on pre-acquired historical execution statistics and the data read performance parameters.

5. The method according to claim 1, wherein, The step of sending the data segmentation and allocation information to the corresponding execution node to perform data reading and query operations includes: generating a target execution node identifier based on the data segmentation and allocation information; determining available execution node communication endpoints based on the target execution node identifier; establishing a bidirectional communication channel based on the execution node communication endpoints; asynchronously sending the data segmentation and allocation information to the corresponding execution node through the bidirectional communication channel, and asynchronously receiving allocation confirmation information returned by the execution node through the bidirectional communication channel; in response to the query coordination node receiving the allocation confirmation information, extracting the data segmentation identifier to be updated and the target status flag based on the allocation confirmation information; locking the corresponding data segmentation record in the data segmentation cache structure based on the data segmentation identifier to be updated; atomically updating the status of the data segmentation record to the target status flag, and releasing the data segmentation record; and sending a data operation execution command to the corresponding execution node to perform data reading and query operations.

6. A data partitioning and scheduling method apparatus for batch mode, comprising: The generation and startup unit is configured to generate query execution plan information in response to the query coordination node receiving query request information sent by the client, and to start an asynchronous processing thread on the query coordination node to iteratively read metadata information of the external data source in batch mode. An execution unit is configured to perform the following steps for each iteration of read metadata information: generate a data segment corresponding to the metadata information, and store the corresponding data segment in a data segment cache structure, wherein the data segment stored in the data segment cache structure is not pre-bound to any execution node; in response to the query coordination node receiving a segmentation request information sent by any of the multiple execution nodes, the following dynamic scheduling steps are performed for the segmentation request information: dynamically select a data segment from the data segment cache structure based on the real-time load information of the execution node corresponding to the segmentation request information to generate data segmentation allocation information; send the data segmentation allocation information to the corresponding execution node to perform data reading and query operations; and complete the data segmentation scheduling operation corresponding to the metadata information in response to the satisfaction of preset conditions.

7. An electronic device, comprising: One or more processors; A storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images