Concurrent derivation method and device based on data distribution statistics, equipment and medium

Abstract

The application relates to the technical field of distributed databases and discloses a concurrent export method and device based on data distribution statistics, equipment and a medium, which comprises the following steps: receiving an export request for a target data table and establishing a communication connection with a computing node; obtaining a primary key field of the target data table from the computing node and determining the primary key field as a statistical field; determining a statistical parameter and sending a statistical creation instruction to drive each storage node to create data distribution statistical information; obtaining a plurality of data intervals from the computing node; constructing a plurality of export statements according to the data intervals; establishing a plurality of concurrent connections to randomly distribute the export statements to drive each storage node to concurrently execute data retrieval; and receiving the export data through the concurrent connections and storing the export data. According to the application, the export granularity is converted from the storage node to the data interval, and the concurrent connection is randomly scheduled, so that fine-grained parallel data retrieval is realized, the performance bottleneck caused by uneven load and sequential execution is reduced, and the export efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, and in particular to a concurrent export method, apparatus, device, and medium based on data distribution statistics. Background Technology

[0002] As data volumes continue to expand, distributed databases have gradually become the mainstream data storage and processing architecture. These databases typically have compute nodes that receive user requests and translate them into access instructions for multiple storage nodes. In practical applications, large-scale data export is a common requirement. Existing export methods often follow the processing approach of single-machine databases, directly sending the complete export statement to each storage node for execution, and then having the compute nodes aggregate the results. This approach is highly dependent on the internal implementation of the storage nodes, and the export process is clearly coupled with the storage node mechanisms, increasing system complexity and maintenance difficulty.

[0003] Meanwhile, the concurrency granularity of existing export methods is usually based on storage nodes. When the amount of data in a single storage node is large, the export time of that node increases significantly, thus becoming the performance bottleneck of the overall export. Even if there are multiple nodes in the cluster, since each node still processes the exported data serially, it is impossible to further split the task within the node, making it difficult to fully utilize the concurrency capabilities of the distributed architecture for the overall export efficiency.

[0004] Furthermore, in the absence of precise data range segmentation criteria, the allocation of export tasks often does not match the actual distribution of data across nodes, easily leading to unbalanced loads. While some methods using data histograms for interval partitioning have the ability to split tasks according to data distribution, in actual execution, they still export sequentially according to node order, failing to establish true concurrency between nodes, thus limiting export efficiency. Summary of the Invention

[0005] The main objective of this invention is to provide a concurrent export method, apparatus, device, and storage medium based on data distribution statistics. This invention aims to solve the technical problem that existing distributed database export methods still use storage nodes as the concurrency granularity and rely on the sequential execution of nodes. They cannot perform fine-grained task division and cross-node concurrent scheduling based on the actual data distribution, resulting in limited export efficiency and uneven load distribution.

[0006] To achieve the above objectives, the present invention provides a concurrent export method based on data distribution statistics, comprising: Receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database; Obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field; Determine the statistical parameters used to create data distribution statistics; Send a statistics creation instruction to the computing node, and drive each storage node to create data distribution statistics based on the statistics fields and parameters; The computing node obtains multiple data intervals contained in the data distribution statistics fed back by each of the storage nodes; Multiple export statements are constructed based on the data range, and each export statement contains a range filtering condition for a corresponding data range. Establish multiple concurrent connections with the computing node, and randomly distribute the exported statement to the concurrent connections to drive each of the storage nodes to concurrently execute data retrieval; The concurrent connection receives and stores the exported data forwarded by the computing node.

[0007] Furthermore, to achieve the above objectives, the present invention provides a concurrent export device based on data distribution statistics, comprising: The request receiving and connection management module is used to receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database. The primary key field extraction and determination module is used to obtain the primary key field of the target data table from the computing node and determine the primary key field as a statistical field; The statistical parameter generation module is used to determine the statistical parameters used to create data distribution statistics. The statistics creation instruction sending module is used to send statistics creation instructions to the computing node, and drive each storage node to create data distribution statistics information based on the statistics fields and statistics parameters. The data interval acquisition and processing module is used to acquire multiple data intervals contained in the data distribution statistics information fed back by each of the storage nodes from the computing nodes; The export statement construction module is used to construct multiple export statements based on the data range, and each export statement contains a range filtering condition for a corresponding data range. The concurrent connection management and task allocation module is used to establish multiple concurrent connections with the computing node and randomly allocate the exported statement to the concurrent connections for distribution, so as to drive each of the storage nodes to concurrently execute data retrieval. The exported data receiving and storage module is used to receive exported data forwarded by the computing node through the concurrent connection and to store the exported data.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a concurrent export program based on data distribution statistics stored in the memory and executable on the processor, wherein when the concurrent export program based on data distribution statistics is executed by the processor, it implements the steps of the concurrent export method based on data distribution statistics as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a concurrent export program based on data distribution statistics, wherein when the concurrent export program based on data distribution statistics is executed by a processor, it implements the steps of the concurrent export method based on data distribution statistics as described above.

[0010] Beneficial Effects: This invention relates to the field of distributed database technology and discloses a concurrent export method, apparatus, device, and medium based on data distribution statistics. The method includes: receiving an export request for a target data table and establishing a communication connection with a computing node; obtaining the primary key field of the target data table from the computing node and determining it as a statistical field; determining statistical parameters and sending a statistical creation instruction to drive each storage node to create data distribution statistics; obtaining multiple data intervals from the computing node; constructing multiple export statements based on the data intervals; establishing multiple concurrent connections and randomly issuing export statements to drive each storage node to concurrently execute data retrieval; and receiving and storing the exported data through concurrent connections. This invention transforms the export granularity from storage nodes to data intervals and combines this with random scheduling of concurrent connections to achieve fine-grained parallel data retrieval, reducing performance bottlenecks caused by uneven load and sequential execution, and improving export efficiency. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a concurrent export method based on data distribution statistics according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the concurrent export method based on data distribution statistics according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the concurrent export device based on data distribution statistics of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The concurrent export method based on data distribution statistics provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server-side component on the compute node side responds to the client's connection request, establishes a communication connection between the client and the compute node, receives the export request containing the target data table identifier information and completes syntax parsing, determines the compute node's network address and service port number based on the target data table's name string, and completes access authentication. The server-side component retrieves the primary key field of the target data table from the compute node and determines the statistical fields, determines the statistical parameters, and generates statistical creation instructions. It then drives the compute node to concurrently distribute these instructions to each storage node to create data distribution statistics. The compute node aggregates the data distribution statistics from each storage node and outputs multiple data ranges. The server-side component constructs multiple export statements based on the data ranges, each containing range filtering conditions for the corresponding data range. The server-side component establishes multiple concurrent connections with the compute node and randomly assigns the export statements to these connections. The compute node forwards the export statements to the corresponding storage nodes based on the range filtering conditions to drive concurrent data retrieval by each storage node. The compute node aggregates and forwards the exported data, and the client receives and stores the exported data through concurrent connections. This invention achieves fine-grained parallel data retrieval by transforming the export granularity from storage nodes to data ranges and combining it with concurrent connection random scheduling. This reduces performance bottlenecks caused by uneven load distribution and sequential execution, thereby improving export efficiency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the concurrent derivation method based on data distribution statistics provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the concurrent export method based on data distribution statistics proposed in this invention includes the following steps: S10, Receive the export request for the target data table and establish a communication connection with the computing node of the distributed database; In this embodiment, receiving an export request for a target data table and establishing a communication connection with the computing nodes of the distributed database involves the collaboration of four types of objects: export request, target data table, computing nodes, and communication connection. The export request refers to the input carrier of the export action, originating from an export command submitted by the user or a set of export parameters submitted by the interface caller. The export request must at least contain the identification information of the target data table to limit the export scope. Receiving the export request corresponds to capturing and reassembling network input data. Capture can be achieved through network port listening, and reassembly can be achieved through buffer aggregation according to message boundaries. After aggregation, a parsable request object is formed. After the request object enters the parsing stage, the identification information of the target data table is obtained through field extraction. The identification information of the target data table can be expressed as a table name string, table identifier, or mapping key-value pair. Field extraction can be achieved through string matching, structured field decoding, or key-value pair parsing. The extraction result is stored in memory as the location basis for the target data table.

[0016] A compute node refers to a node entity in a distributed database that handles user request parsing and task scheduling. The network address and service port number of the compute node serve as the connection parameters for establishing a communication connection. To locate the compute node from the target data table, a node mapping relationship can be set up. This mapping relationship stores the correspondence between the target data table and the compute nodes. The node mapping relationship can originate from a local configuration file, a centralized configuration center, or the database metadata directory. Based on the identifier information of the target data table, the node mapping relationship is queried to obtain the network address and service port number of the compute node. The query can be implemented using index retrieval or hash lookup, and the retrieval results are stored in memory as connection target information.

[0017] A communication connection refers to the transmission channel that maintains data exchange between the server and the compute node. Establishing a communication connection involves a connection handshake and authentication interaction. The connection handshake is initiated by the database connection driver, which is responsible for converting the network address and service port number into a low-level connection request and submitting access authentication credentials along with the handshake request to the compute node. Access authentication credentials originate from permission configuration, session tokens, or credential data issued by an authentication component. After the compute node returns a successful authentication response signal, the communication connection is registered in the connection management structure. The connection management structure records the connection status, connection handle, and context identifier associated with the export request, so that the communication connection can be reused in subsequent data interactions. Thus, the export request completes a closed loop from input reception, target data table location, compute node location, to communication connection establishment, and each object has a clear source and an executable processing path.

[0018] Multiplexed network listening can be used to receive export requests from multiple sources on a single port, distinguishing different requests by session identifiers and aggregating request content in a buffer according to session dimensions. Alternatively, independent ports can be allocated to different callers, and independent request buffers can be configured for each port to reduce data interference across callers. Node mapping relationships can adopt a static loading mode, reading node mapping relationships and caching them in memory when the process starts to reduce query overhead; or a dynamic refresh mode, pulling node mapping relationships from the configuration center at time intervals and updating the memory cache to adapt to the scaling of compute nodes. Communication connections can adopt a connection pool mode, maintaining a fixed number of communication connections for each compute node and distributing requests according to load; or an on-demand creation mode, creating communication connections after receiving export requests and releasing connections after the request ends to adapt to short-term bursts of requests. Access authentication credentials can adopt a fixed credential mode, with credentials loaded by the configuration; or a temporary credential mode, with credentials issued by the authentication component and having an expiration date set before each communication connection is established.

[0019] This embodiment can reliably obtain the identification information of the target data table by receiving and parsing the export request in a structured manner, and determine the network address and service port number of the computing node based on the node mapping relationship. Combined with the database connection driver, it completes the handshake and authentication, thereby forming a reusable communication connection, reducing the probability of export request failure due to unclear computing node location or incomplete connection establishment, and reducing the connection overhead caused by repeated connection establishment.

[0020] S20, obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field; In this embodiment, the computing node, as the entry point for request carrying and metadata access, needs to have the ability to read the target data table structure information. The structure information can come from the system directory, data dictionary, or metadata cache. The target data table is used to limit the scope of metadata access. The identification information of the target data table is already available before the start of this step, and the computing node locates the corresponding table structure entry based on this identification information.

[0021] Retrieving the primary key field corresponds to a metadata retrieval and parsing process. Metadata retrieval involves the compute node submitting a structure retrieval request related to the target data table. This request can be expressed using a structured query message or an internal database metadata access interface, carrying the target data table's identification information. The compute node returns table structure definition information, which includes at least a set of field definitions and a set of constraint definitions. The field definition set includes field names, field types, field order, or field identifiers; the constraint definition set includes constraint type identifiers for primary key constraints and a set of associated field identifiers. The parsing process breaks down the table structure definition information into processable data items and filters entries in the constraint definition set whose constraint type identifiers are primary key constraints. This yields the set of field identifiers associated with the primary key constraint, which is then mapped back to the field definition set to obtain the primary key field.

[0022] A primary key field can be a single-field primary key or a composite primary key. A single-field primary key corresponds to a set of field identifiers containing only one field identifier. A composite primary key corresponds to a set of field identifiers containing multiple field identifiers, and a primary key field for statistical analysis needs to be determined from these multiple field identifiers. The determination rule needs to be executable and repeatable. A common implementation is to select the field corresponding to the first field identifier in the field definition order, which comes from the field order attribute in the field definition set. Alternatively, it can be sorted by field name or by field identifier, but the selection result must be consistent across multiple executions of the same target data table.

[0023] The primary key field is designated as a statistical field, corresponding to a single field reference binding operation. The statistical field serves as a placeholder for subsequent statistical operations, establishing a clear assignment relationship in the current step. The statistical field can be expressed as a field name string, field identifier, or field path. The determination process copies or maps the expression of the primary key field to the expression of the statistical field and writes the statistical field into the context data structure of the current export task for direct reading in subsequent stages. This binding operation must avoid ambiguity; the relationship information between the statistical field and the target data table should be preserved to prevent confusion between different tables with fields having the same name.

[0024] Retrieval can be performed using a system directory-based method. After receiving a structure retrieval request, the compute node reads the table structure entries from the system directory and returns table structure definition information containing the field definition set and constraint definition set. Alternatively, a metadata caching-based method can be used. The compute node maintains a cache of the target data table's table structure definition information in memory. When a structure retrieval request hits the cache, it directly returns the cached content; otherwise, it reads the system directory and refills the cache. Primary key field identification can use a constraint-first approach, first locating the primary key constraint entries and then backtracking to the field definition set. Alternatively, a field-marking approach can be used, directly reading the flag indicating whether a field belongs to the primary key within the field definition set and aggregating the primary key fields. The selection rule for composite primary keys can use a field definition order-first approach, selecting the field with the lowest order as the primary key field. Alternatively, a field type-first approach can be used, filtering by field type priority within the field identifier set and then determining by field order. Statistical field binding can use a direct assignment approach, writing the primary key field's field name string into the statistical field. Alternatively, an identifier binding approach can be used, writing the primary key field's field identifier into the statistical field and maintaining a mapping table between the identifier and field name in the context.

[0025] This embodiment retrieves and parses the table structure definition information of the target data table on the computing node side, which can reliably obtain the primary key field associated with the primary key constraint. When a composite primary key exists, a unique primary key field is determined through a repeatable selection rule. Then, a clear binding relationship is established between the primary key field and the statistical field, so that the statistical field has a clear source and consistent expression, reducing statistical input ambiguity caused by uncertain field selection.

[0026] S30, Determine the statistical parameters used to create data distribution statistics; In this embodiment, the statistical parameters used to create data distribution statistics are determined, involving the collaboration between objects such as statistical parameters, data distribution statistics, statistical fields, computing nodes, and communication connections. Statistical parameters define the granularity, scope, and output structure of the data distribution analysis, serving as the input basis for driving the subsequent generation of data distribution statistics. The source of statistical parameters can be determined by control information carried in the export request, pre-set configuration items within the system, or dynamic configurations in the runtime environment. Statistical fields are the core objects of data distribution analysis, and statistical parameters are set around the value range of these statistical fields.

[0027] Statistical parameters include the number of buckets controlling the precision of interval division, the sampling ratio limiting the statistical range, the statistical mode specifying the statistical method, and the output format describing the result structure. The number of buckets determines how many numerical intervals the statistical field is divided into, directly affecting the density of data intervals in the data distribution statistics. The sampling ratio reduces the amount of statistical computation when dealing with large datasets, deriving the overall distribution trend through sampling analysis of the statistical field. The statistical mode specifies whether to use exact or approximate statistical methods; exact statistics iterate through all data records, while approximate statistics extrapolate based on sampling results. The output format standardizes the organization of the data distribution statistics, facilitating subsequent parsing and interval combination.

[0028] Statistical parameters are stored in memory as structured data, with each parameter item represented by a key-value pair structure. The parameter determination process either reads configuration items from the export request or loads default values ​​from the system's preset configuration. When no corresponding parameter is provided in the export request, the system's preset default parameter is used. After the parameters are determined, the statistical parameters are associated with statistical fields, explicitly pointing the statistical parameters to the statistical fields. The statistical parameters then serve as input elements for generating statistical creation instructions in subsequent processing.

[0029] Parameter configuration items can be parsed from the export request to generate a statistical parameter structure. Alternatively, default statistical parameters can be read from the system configuration file and then overridden based on runtime conditions. The number of buckets can be dynamically adjusted according to the data size of the target data table, the sampling ratio can be dynamically adjusted according to the resource status of the compute nodes, and the statistical mode can be selected according to accuracy requirements. Statistical parameters can be stored in the session context or in a temporary memory area for later retrieval.

[0030] This embodiment predetermines the statistical parameters associated with the statistical fields, enabling the generation of data distribution statistics with controllable precision and structure, avoiding problems such as inconsistent statistical granularity or uninterpretable statistical results, and providing a stable data foundation for subsequent interval division.

[0031] S40, send a statistics creation instruction to the computing node, and drive each storage node to create data distribution statistics information based on the statistics fields and statistics parameters; In this embodiment, sending a statistical creation command to the computing node corresponds to a control message generation and delivery process directed to the computing node. The statistical creation command serves as a request carrier to create statistical information on data distribution. The content of the statistical creation command includes at least two types of inputs: statistical fields and statistical parameters. Statistical fields point to the data columns being statistically analyzed, while statistical parameters indicate the control quantities for the statistical granularity and range. The statistical creation command needs to have a parsable structural representation, which can be a key-value pair message, a structured protocol message, or a database internal command object. Statistical fields within the structure can be expressed as field name strings, field identifiers, or field paths, while statistical parameters can be expressed as numerical parameters, interval parameters, or bucket parameters. The statistical creation command also needs to carry the association identification information of the target data table so that the computing node can bind the statistical fields to the specific table structure, avoiding ambiguity in field referencing in multi-table scenarios.

[0032] Based on statistical fields and parameters, when a compute node parses statistical creation instructions, it resolves statistical fields into executable field references and statistical parameters into executable statistical control variables. Field reference resolution includes field existence verification, field type reading, and field encoding method determination. Field type reading determines the boundary comparison method and value normalization method on the storage node side, while field encoding method determines the read path of the statistical field in the storage layer. Statistical parameter resolution includes parameter validity verification and parameter normalization. Validity verification constrains the parameter value range and value type, while normalization converts statistical parameters into a unified expression that the storage node can directly use, such as converting interval granularity parameters into bucket numbers or bucket numbers into interval boundary generation rules.

[0033] The creation of data distribution statistics by compute nodes driving storage nodes corresponds to a task splitting and distribution process directed to storage nodes. After obtaining the executable representations of statistical fields and parameters, the compute node needs to determine the set of storage nodes containing the data to be created. This set can be derived from the distribution metadata of the target data table. The compute node generates multiple subtask descriptions for the storage node set. Each subtask description includes at least an execution reference to the statistical field, the execution value of the statistical parameter, and the shard identifier of the target data table on the corresponding storage node. The compute node sends the subtask descriptions to each storage node, enabling each storage node to read the data records related to the statistical fields in its local storage layer and generate data distribution statistics according to the rules defined by the statistical parameters. The representation of the data distribution statistics can include bucket boundary values ​​and bucket counts, or interval start boundary values, interval end boundary values, and interval counts. The representation format needs to be retrievable and aggregated by the compute node, but in this step, the focus is on the creation action being triggered and the creation basis clearly derived from the statistical fields and parameters.

[0034] The delivery of statistical creation commands relies on a communication connection, which carries the transmission of statistical creation commands from the initiator to the computing node. The delivery process includes three stages: command serialization, transmission and confirmation. Command serialization converts the statistical creation command into a transmittable byte sequence. Transmission writes the byte sequence into the transmission buffer of the communication connection. Confirmation is achieved when the computing node returns a receipt confirmation message or a receipt status code. The confirmation content indicates that the computing node has entered the parsing and driving stage, ensuring that the statistical creation command is not lost at the transport layer, thus preventing the storage node from failing to receive the creation trigger.

[0035] Structured protocol messages can be used as the structural representation of statistical creation instructions. The target data table identifier, statistical fields, and statistical parameters are encoded into a fixed set of fields, which is transmitted to the compute node via a communication connection. The compute node then parses the data based on this fixed set of fields. Alternatively, a command object approach can be used. A database command object is constructed at the initiating end and submitted to the compute node through the driver interface. The compute node's internal command executor then extracts the statistical fields and parameters. The storage node set can be determined using distributed metadata queries, where the compute node reads the mapping table from the target data table's shards to the storage nodes to obtain the storage node set. Alternatively, a routing caching approach can be used, where the compute node maintains a mapping table cache and reads directly within the cache's validity period. Subtask descriptions can use a unified task description structure. Each storage node receives a subtask description containing field identifiers for the statistical fields and normalized values ​​for the statistical parameters. The storage node locates the column storage file or row storage page based on the field identifiers and generates bucket boundary values ​​based on the normalized values. Arrival confirmation can be achieved synchronously, where the compute node returns a confirmation message before proceeding to the driver, or asynchronously, where the compute node returns an acceptance flag and schedules the driver task in its internal queue.

[0036] This embodiment encapsulates statistical fields and parameters into a statistical creation instruction and delivers it to the computing node via a communication connection. After parsing, the computing node can form a consistent execution reference and consistent parameter expression for each storage node, and split the creation request into a sub-task description for each storage node. This allows each storage node to generate data distribution statistics under the same statistical fields and the same statistical parameter constraints, reducing the deviation of statistical results caused by inconsistent statistical standards.

[0037] S50, obtain multiple data intervals contained in the data distribution statistics information fed back by each of the storage nodes from the computing nodes; In this embodiment, the computing node corresponds to the execution-side node in the distributed database that handles requests and aggregates scheduling. The "acquisition" action corresponds to reading the aggregated statistical results from the computing node. This reading can be done through the existing communication connection's receiving channel or through a dedicated result retrieval interface. The storage node feedback corresponds to the storage node sending back statistical results to the computing node after generating data distribution statistics locally. The feedback carrier can be a structured message, a binary message, or a tabular result set. The computing node aggregates the feedback from multiple storage nodes into a readable aggregated result. The data distribution statistics correspond to a distribution description set formed around the statistical fields. This distribution description set contains at least elements that can deduce interval boundaries. Boundary elements can be represented as bucket boundary value sequences, interval start and end boundary value pairs, or quantile sequences arranged by ordered keys. Multiple data intervals correspond to a set of intervals extracted and organized from the data distribution statistics. Each interval in the set has two endpoints: a start boundary and an end boundary. The endpoint values ​​come from the boundary elements of the data distribution statistics. The number of interval sets is related to the statistical parameters, and the order of the interval sets is related to the sorting rules of the statistical fields.

[0038] The process of retrieving multiple data intervals from a computing node involves four actions: result location, result reception, structure parsing, and interval extraction. Result location identifies readable statistical result resource identifiers on the computing node side. These resource identifiers can be a combination of a statistical task identifier, a target data table identifier, and a statistical field identifier, or an acceptance identifier returned by the computing node. Result reception reads the statistical result content corresponding to the resource identifier from the computing node. This process handles fragmented packet return and packet reassembly / disassembly to ensure the integrity of the statistical result content. Structure parsing restores the statistical result content to the internal structure of the data distribution statistics. This internal structure must retain the correspondence between storage node identifiers and boundary elements to distinguish different storage node sources during interval extraction. Interval extraction generates multiple data intervals from boundary elements. Generation rules can involve pairing adjacent bucket boundary values ​​to form interval endpoints, or directly reading the interval start and end boundary values ​​as interval endpoints. After interval extraction, a set of multiple data intervals is formed, which can be an array of interval objects, an interval endpoint table, or a serialized interval list.

[0039] The accuracy of multiple data intervals depends on the orderliness and consistency control of boundary elements. Orderliness requires boundary elements to be arranged according to the sorting rules of statistical fields, which can be numerical ascending order, lexicographical order, or time order. Consistency control requires that the data distribution statistics returned by the same storage node maintain the same caliber in terms of statistical fields and statistical parameters as other storage nodes. The summary results on the computing node side need to include statistical field identifiers and statistical parameter values. The initiating end performs consistency checks using statistical field identifiers and statistical parameter values ​​during parsing to avoid the generation of multiple erroneous data intervals due to the mixing of boundary elements with different statistical calibers.

[0040] The data can be retrieved using either a push-based method, where the computing node sends the summarized statistical results data via a communication connection after aggregation, and the receiving end matches the corresponding summarized statistical results data based on the statistical task identifier and completes parsing and interval extraction. Alternatively, a pull-based method can be used, where the receiving end initiates a result pull request to the computing node, carrying the target data table identifier and statistical field identifier. The computing node returns a result set containing data distribution statistics, and the receiving end parses the result set to generate multiple data intervals. Interval extraction can be performed using a boundary sequence pair generation method, combining adjacent boundary values ​​in the bucket boundary value sequence to form the start and end boundaries, or using an explicit interval reading method, directly reading the interval start and end boundary value fields to construct interval objects. Structure parsing can be performed using a protocol decoding method, decoding the binary message according to field definitions into a set of data distribution statistics objects, or using a table mapping method, mapping the boundary fields in the result set to an array of boundary elements while retaining the storage node identifier field to distinguish the source.

[0041] This embodiment reads the aggregated data distribution statistics from the computing node and parses and extracts multiple data intervals at the reading end. This can transform the boundary elements returned by multiple storage nodes into a unified interval set expression, reducing the sensitivity to differences in representation within storage nodes during interval generation and reducing the risk of interval missing or overlapping caused by interval endpoint mismatch.

[0042] S60, construct multiple export statements based on the data range, each export statement containing a corresponding range filtering condition for the data range; In this embodiment, the data intervals correspond to the interval set elements extracted from data distribution statistics. Each data interval contains at least two types of boundary information: a start boundary and an end boundary. The value type of the boundary information is consistent with that of the statistical field. When the statistical field is a numeric type, the boundary information is a numeric value; when the statistical field is a character type, the boundary information is a character that can be compared in lexicographical order; and when the statistical field is a time type, the boundary information is a time point. Multiple export statements are constructed to map each of the multiple data intervals into a data retrieval expression that can be parsed by the computation node and executed by the storage node. The export statements, as executable statement text or serializable statement objects, contain at least three types of content: target data table identifier, field selection expression, and filtering expression. The interval filtering conditions correspond to the interval limitation part in the filtering expression. The interval limitation part uses the statistical field as the comparison object and the start and end boundaries as comparison boundaries, thus restricting the value range of the statistical field.

[0043] Generating multiple export statements requires five steps: defining the statement framework, reading interval boundaries, generating condition expressions, assembling statements, and outputting them as a set. Defining the statement framework clarifies the basic structure of the export statements. This basic structure can be either a query statement structure or an export statement structure. It reserves space for inserting interval filtering conditions and specifies the target data table. Reading interval boundaries retrieves the start and end boundaries of each data interval and converts them into literal forms compatible with the export statement expressions. These literal forms can be numeric, string, or time literals. Generating condition expressions generates interval filtering conditions based on statistical fields, start and end boundaries. Each interval filtering condition must contain at least two comparison clauses: a lower bound comparison and an upper bound comparison. The lower bound comparison associates the statistical field with the start boundary, and the upper bound comparison associates the statistical field with the end boundary. Comparison operators can be combinations of greater than or equal to and less than to control boundary coverage, or combinations of greater than and less than or equal to. Interval filtering conditions can use logical AND to connect two comparison clauses to form closed or half-open interval expressions. Statement assembly is used to fill the filter conditions of the range into the filter expression positions of the exported statement, and to complete the target data table and field selection expression. The field selection expression can be a full field export or a field subset export. Field subset export requires specifying the exported column set through the field list. Set-based output is used to aggregate the exported statements generated for each data range into an exported statement set. The exported statement set can be a statement array, a statement queue, or a statement mapping table with range indexes. Each exported statement in the exported statement set maintains a one-to-one correspondence with its corresponding data range. The exported statement set needs to retain traceable information of this one-to-one correspondence, such as range identifiers or range indexes.

[0044] The correctness of interval filtering conditions depends on boundary consistency and boundary coverage control. Boundary consistency requires that the starting boundary be less than or equal to the ending boundary, that the types of the starting and ending boundaries be consistent with the statistical fields, and that the formats of the starting and ending boundaries meet the requirements of the exported statement parsing. Boundary coverage control is used to avoid duplication or omission at the boundaries of adjacent data intervals. When the interval filtering conditions are expressed using half-open intervals, the shared boundaries between adjacent intervals will not lead to duplicate records. When the intervals are expressed using closed intervals, the boundaries of adjacent intervals need to be offset or a sort key deduplication rule needs to be used to avoid duplicate records.

[0045] A template assembly approach can be used to maintain a basic structure template for the exported statements. This template includes the target data table location and the filter expression location. The start and end boundaries of each data interval are read sequentially to generate interval filter conditions. These conditions are then written into the filter expression location of the basic structure template, and the exported statement text is output. Alternatively, an abstract syntax tree approach can be used. The target data table, statistical fields, start and end boundaries are constructed into statement object tree nodes, and these nodes are serialized to obtain the exported statements. Condition expression generation can employ a half-open interval strategy, setting the lower bound comparison to the statistical field being greater than or equal to the start boundary and the upper bound comparison to the statistical field being less than the end boundary. A closed interval strategy can also be used, setting the lower bound comparison to the statistical field being greater than or equal to the start boundary and the upper bound comparison to the statistical field being less than or equal to the end boundary, with non-overlapping adjustments made to adjacent boundaries during interval boundary generation. Set-based output can be achieved by appending statements sequentially according to interval index, adding each exported statement to the exported statement set in turn, or by grouping statements by storage node, grouping them by data interval source and then adding them to the exported statement set to adapt to different scheduling strategies.

[0046] This embodiment maps each data interval to an export statement containing the corresponding interval filtering conditions and aggregates them into multiple export statements. This allows the interval boundary constraints to be explicitly fixed into the export statements, forming a stable one-to-one correspondence between the export statements and the data intervals, and reducing the risk of range mismatch caused by inconsistent interval boundary expressions.

[0047] S70, establish multiple concurrent connections with the computing node, and randomly distribute the exported statement to the concurrent connections to drive each of the storage nodes to concurrently execute data retrieval; In this embodiment, multiple concurrent connections correspond to multiple independent communication connection instances maintained within the same time window. The peer of a concurrent connection is a computing node, and the local end of a concurrent connection is an export tool or an execution entity that initiates an export request. The independence of concurrent connections is reflected in the mutual isolation of connection identifiers, session states, send / receive buffers, and flow control states. The number of concurrent connections is a configurable or negotiable number. Establishing multiple concurrent connections with the computing node corresponds to a set of connection creation actions. The set of connection creation actions includes four types of actions: connection parameter preparation, connection establishment request sending, connection establishment confirmation reception, and connection status registration. Connection parameter preparation includes the computing node's network address and service port number, access authentication credentials, and connection attribute information. Connection attribute information includes connection timeout parameters, maximum message length parameters, and a set of concurrent connection identifiers. Connection status registration is used to add each concurrent connection to a connection management structure. The connection management structure can be a connection table or a connection queue, and it records the connection identifiers and busy status of concurrent connections.

[0048] The exported statement corresponds to the executable statement text or serializable statement object generated from the data range. The minimum content of the exported statement is the target data table identifier and the range filtering conditions, which are used to limit the data retrieval range. Random allocation corresponds to the nondeterministic mapping of exported statements to concurrent connections. The nondeterministic mapping is generated by a random source, which can be a pseudo-random number generator or a hardware random source. The random index is used to select exported statements or concurrent connections, and the random sequence is used to shuffle the set of exported statements or rotate the set of concurrent connections. Random allocation needs to satisfy balance constraints and controllability constraints. Balance constraints are used to prevent a certain concurrent connection from carrying too many exported statements for a long time. Controllability constraints are used to ensure that exported statements are not repeatedly allocated and that exported statements are fully allocated. The non-repeated allocation of exported statements depends on task status flags, which include three states: unallocated, allocated, and completed.

[0049] The "Deployment" action sends the exported statement to the compute node via the selected concurrent connection. This sending action includes three sub-actions: message encapsulation, message transmission, and transmission acknowledgment. Message encapsulation combines the exported statement with necessary context information to form the transmission payload. The context information includes the exported statement identifier, range identifier, and retry identifier. Transmission acknowledgment updates the busy status of the concurrent connection and the task status flag of the exported statement. The "Driven" action, which involves the compute node parsing the exported statement and routing it to the target storage node, triggers concurrent execution. This means the compute node transforms the exported statement into an executable retrieval request for the storage node and issues an execution trigger. Concurrent execution by storage nodes means multiple storage nodes are simultaneously in data retrieval execution state within an overlapping time window. The "Data Retrieval" action, which involves performing data reading operations at the local storage layer of the storage node, limited by range filtering conditions. This includes three types of actions: index positioning or full table scan, record filtering, and result output. Index positioning locates the candidate record set based on the index structure of statistical fields. Full table scan is used to traverse the candidate record set when no available index is available. Record filtering removes records that do not meet the range filtering conditions.

[0050] The operation of concurrent connections requires connection state management and congestion control. Connection state management includes busy state updates and idle state releases. A busy state indicates that there are unfinished exported statements being processed on the concurrent connection, while an idle state indicates that the concurrent connection can handle the delivery of new exported statements. Congestion control includes send window control and retry control. Send window control limits the number of exported statements in transit for a single concurrent connection, while retry control reallocates concurrent connections for exported statements that fail to be sent or time out. Retry control relies on retry flags to avoid infinite retries.

[0051] A fixed number of concurrent connections can be used, establishing multiple concurrent connections based on a preset parallelism configuration. Connection identifiers and initial idle states are registered in the connection management structure. Random allocation uses a random index to select export statements from the export statement set and concurrent connections from the idle concurrent connection set. The export statements are encapsulated as a transmission load and sent to the compute node via the selected concurrent connection, which is then marked as busy. Alternatively, a dynamic number of concurrent connections can be used, adjusting the number based on the connection capacity parameters returned by the compute node. Within the maximum number of concurrent connections, connections are established as needed, and long-idle connections are reclaimed. Random allocation involves shuffling the export statement set and selecting concurrent connections in a round-robin manner to improve load balancing. A windowed delivery method can also be used, setting a maximum number of export statements in transit for each concurrent connection. Sending confirmation is based on the receiving confirmation returned by the compute node, and when the maximum number of in-transit statements is reached, delivery to that concurrent connection is paused, and the process redirects to other idle concurrent connections. The data retrieval trigger can carry a range identifier in the transmission load, allowing the compute node to retain the range identifier during export statement parsing for routing and tracing.

[0052] This embodiment establishes multiple concurrent connections and randomly distributes exported statements to these connections, providing multiple parallel distribution channels at the connection level. Random distribution reduces the risk of concentrated exported statements across concurrent connections. Furthermore, by using busy status and task status flags, it achieves complete distribution and load balancing of exported statements, thereby increasing the probability and coverage of storage nodes entering concurrent data retrieval execution state.

[0053] S80, receive and store the exported data forwarded by the computing node through the concurrent connection.

[0054] In this embodiment, concurrent connections correspond to established communication connection instances that are ready to send and receive. Each concurrent connection has an independent receive buffer, connection identifier, and session state. Concurrent connections are used to carry data packets forwarded by the compute nodes. Receiving corresponds to listening for and reading arriving data packets within the receive buffer of the concurrent connection. The source of the data packets is the aggregation and forwarding result of the compute nodes' retrieval of results from the storage nodes. The content of the data packets is a byte sequence or character sequence of the exported data. During the transmission phase, the exported data is represented as a continuous data stream or fragmented data blocks. At the receiving end, the fragmented data blocks need to be spliced ​​and restored based on packet boundary information or length information. The splicing process relies on the cache management structure in the receive buffer, which is used to temporarily store incompletely spliced ​​data fragments.

[0055] The storage process involves writing the received and restored exported data to a persistent storage area. This persistent storage area can be a file object in a file system, an object entity in object storage, or a storage block in a block device. The write operation relies on a memory buffer, which temporarily stores the assembled exported data in memory. The memory buffer has a capacity threshold and a refresh policy. The capacity threshold limits the amount of data that can be stored in the memory buffer, and the refresh policy triggers the write operation when the capacity reaches the threshold or when a signal indicating the end of reception arrives. The write operation includes data serialization, a write interface call, and write result confirmation. Data serialization converts the exported data in memory into a writable format according to a predetermined format. The write interface call writes the serialized data to the persistent storage area, and the write result confirmation determines whether the write was successful and updates the buffer state.

[0056] After being received, the exported data may require formatting. This formatting process depends on the row and column delimiter configuration. The row and column delimiter configuration is used to parse the continuous byte stream into structured record units. Each record unit consists of a set of fields, which are then split according to the delimiter rules. The split record units are reorganized into structured record data, which is then entered into a memory buffer for subsequent writing. Once storage is complete, an export file is generated. The export file corresponds to a file entity or object entity in the persistent storage area containing the complete exported data.

[0057] An event-based receiving method can be used, registering a receive listener on each concurrent connection. Upon receiving a data packet, it is immediately read and written to the receive buffer. The fragmented data blocks in the receive buffer are then concatenated to restore the complete exported data before entering the memory buffer. A write operation is triggered when the data volume in the memory buffer reaches a capacity threshold. Alternatively, a polling method can be used to periodically read the receive buffers of concurrent connections, concatenate and format the read data sequentially, and then write it to the memory buffer. A forced flush of the memory buffer is performed upon receiving an end-of-time marker to complete the write. A streaming write method can also be used, directly writing to the persistent storage area in a streaming manner after concatenation and formatting, and using an internal counting mechanism instead of a capacity threshold for judgment.

[0058] This embodiment continuously receives exported data forwarded by computing nodes on concurrent connections and combines buffering, splicing, formatting, and write control to ensure that exported data can be continuously and completely restored and reliably written to the persistent storage area after transmission is completed. This avoids data loss or disorder due to fragmented transmission and improves the integrity and stability of exported data.

[0059] In one embodiment, step S10 includes: S101, listen to the instruction data stream input from the user interaction interface, and capture the export request containing the identification information of the target data table; S102, perform syntax parsing on the export request to extract the table name string of the target data table; S103, based on the table name string of the target data table, query the node mapping relationship to determine the network address and service port number of the corresponding computing node; S104, Invoke the database connection driver, initiate a network connection handshake with the computing node based on the network address and the service port number, and submit access authentication credentials; S105, after the computing node returns an authentication successful response signal, confirm the establishment of a communication connection with the computing node.

[0060] In this embodiment, the user interaction interface is used to carry the input and submission of the export operation. The user interaction interface can be a graphical interface control, a command line input channel, a remote call entry point, or a message subscription entry point. The user interaction interface exposes the input endpoints and parameter constraint rules for submitting export requests. The instruction data stream corresponds to the character sequence, structured message, or event sequence continuously entering from the user interaction interface. The instruction data stream includes a request type identifier, the identifier information of the target data table, and a set of optional parameters related to the export. Listening to the instruction data stream input from the user interaction interface corresponds to establishing an event listening or buffer reading mechanism on the input endpoint. The listening process includes receiving input events, aggregating input fragments, validating input validity, and determining the triggering capture conditions. The capture conditions are used to identify input fragments in the instruction data stream that meet the export request format. Capturing the export request containing the identifier information of the target data table corresponds to solidifying the input fragments that meet the capture conditions into an export request object. The export request object includes the identifier information of the target data table, the submission timestamp, the request source identifier, and the request parameter set. The export request object serves as the input carrier for subsequent parsing and connection establishment, avoiding the repeated processing of the same instruction data stream.

[0061] Syntax parsing is used to parse expressions in the export request object into a computable, structured representation. Syntax parsing can employ a combination of lexical segmentation, syntax tree construction, and semantic verification. Lexical segmentation divides the character sequence in the export request into a token sequence, which includes keyword tokens, delimiter tokens, identifier tokens, and constant tokens. Syntax tree construction generates syntax tree nodes based on the token sequences. These nodes represent table reference locations, parameter locations, and constraint locations in the export request. Semantic verification checks the syntax tree nodes for rule constraints, including whether the target table identifier exists, whether the identifier conforms to naming rules, and whether the parameter set satisfies type and value range constraints. Extracting the table name string of the target table involves locating the table reference node in the syntax tree and reading the table identifier content. The table name string serves as a stable addressing key and is stored in the export request object or an independent field variable. The table name string must be unique and comparable to avoid mapping failures caused by name ambiguity or inconsistent formats during subsequent addressing.

[0062] Node mapping relationships describe the correspondence between target data tables and compute nodes. These relationships can exist in the form of mapping tables, routing tables, metadata directories, or service discovery records. A node mapping relationship must at least contain an associated entry with a table name string and a compute node identifier. This associated entry may further include the network address, service port number, and connection protocol type. Querying a node mapping relationship involves using the table name string as the query key to perform a lookup operation. This lookup operation can be a key-value index lookup, directory search, or cache hit lookup. The lookup result is used to determine the network address and service port number of the corresponding compute node. The network address indicates the reachable network location of the compute node and can be a domain name, IP address, or a resolvable service name. The service port number indicates the port entry point where the compute node provides connection services. The service port number and network address together constitute the connection endpoint parameters. The determination process includes verification of the resolvability of the network address and the numerical range of the service port number to prevent connection establishment failures and duplicate requests caused by unreachable addresses or invalid ports.

[0063] The database connection driver is used to initiate a network connection handshake with the compute node and manage the connection session state. The database connection driver can encapsulate transport layer connection establishment, application layer protocol negotiation, authentication message encapsulation, and error code parsing. Calling the database connection driver corresponds to passing the network address and service port number to the driver and triggering the connection establishment process, which includes socket creation, connection request sending, handshake message exchange, and session parameter negotiation. The network connection handshake is used to negotiate the communication protocol version, encryption parameters, and session identifier between the client and the compute node. The output of the network connection handshake is the session state and connection identifier that can be used for subsequent data interaction. Access authentication credentials are used to complete identity verification and access authorization. Access authentication credentials can be password digests, token strings, certificate chains, or signed messages. Submitting access authentication credentials corresponds to encapsulating the access authentication credentials in an authentication message and sending it to the compute node during the handshake phase or the authentication phase after the handshake. The authentication message is bound to the session state to prevent authentication confusion caused by cross-session reuse.

[0064] The response signal characterizes the compute node's verification result of the access authentication credentials. The response signal can be an explicit authentication pass flag, status code, and an accompanying set of session parameters. The compute node returns an authentication pass response signal corresponding to an acknowledgment message returned after the handshake and authentication process. This acknowledgment message contains the authentication result and session validity information. Confirming the establishment of a communication connection with the compute node corresponds to updating the local connection status to available upon receiving the authentication pass response signal and registering the communication connection as a session resource associated with the table name string. The registration action includes recording the connection identifier, session parameters, timeout configuration, and error recovery strategy to ensure that subsequent interaction requests for the same target data table can reuse the communication connection and be executed in a consistent session context.

[0065] This embodiment forms a traceable export request object by listening to and capturing the instruction data stream, obtains a stable table name string based on syntax parsing, determines the network address and service port number using node mapping relationships, and completes the network connection handshake and access authentication credential submission in conjunction with the database connection driver, confirming the availability of the communication connection with a response signal. This ensures that the process from export request to communication connection establishment has a clear input-output relationship and verifiable intermediate results, reducing connection failures and repeated connection establishment caused by incomplete input, inconsistent addressing, or uncertain authentication status, and improving the stability and repeatability of connection establishment.

[0066] In one embodiment, step S20 above includes: S201, Send a metadata query request to the computing node through the communication connection; S202, Receive the table structure definition information of the target data table returned by the computing node; S203, Parse the constraint attribute data from the table structure definition information; S204, Identify the primary key field with a primary key constraint identifier based on the constraint attribute data; S205, determine whether the identified primary key field is a composite primary key containing multiple fields; S206, If the primary key field is a composite primary key, then the first field in the composite primary key is selected as the final determined primary key field according to the field definition order; S207, if the primary key field is not a composite primary key, then the identified primary key field will be used as the final determined primary key field; S208, the final determined primary key field is used as the statistical field.

[0067] In this embodiment, the communication connection is used to carry out bidirectional message interaction with the computing node. The communication connection includes connection attributes such as session identifier, transmission protocol parameters, timeout configuration, and retry policy. The existence of the communication connection enables metadata query requests to be sent to the target computing node and receive corresponding responses. Sending a metadata query request to the computing node is used to obtain structure-related metadata of the target data table. The content of the metadata query request includes the identification information of the target data table and the query type identifier. The query type identifier indicates that the returned content covers field definitions, constraint definitions, or a combination of both. The sending process of the metadata query request includes request message construction, request message encoding, and request message delivery. During message construction, the identification information of the target data table is written into the query parameter area, and the query type identifier is written into the control field area. During message encoding, a serialized byte sequence is generated according to the connection protocol. During message delivery, the message is submitted through the sending buffer of the communication connection and waits for confirmation of receipt from the computing node.

[0068] Table structure definition information describes the structure and constraint set of the target data table. It includes at least a field definition set and a constraint definition set. The field definition set contains field attributes such as field name, field type, whether nullables are allowed, default value, and field definition order. The field definition order is used for order determination in subsequent composite primary key processing. The constraint definition set contains constraint attributes such as constraint type, constraint name, participating field set, and constraint attribute identifier. The constraint attribute identifier distinguishes different constraint categories such as primary key constraints, unique constraints, and foreign key constraints. Receiving the table structure definition information returned by the compute node corresponds to reading and decoding the response message in the communication connection's receive buffer. After decoding, the table structure definition information object is obtained and stored locally in a structured form. This structured form can be a key-value mapping, a tree structure, or a table structure, serving as stable input for subsequent parsing operations.

[0069] Constraint attribute data is parsed from the table structure definition information to separate constraint-related content from the structured object. Constraint attribute data is the extracted result of the constraint definition set, retaining the association relationship with the fields. The parsing process includes locating the constraint definition set, traversing each constraint definition, reading the constraint type and participating field set, and generating constraint attribute data entries. Each constraint attribute data entry contains fields such as constraint type identifier, primary key constraint identifier, and participating field set. To support subsequent identification operations, the constraint attribute data can also contain field definition order information for the participating field set or mapping information from field names to field definition order, enabling order selection locally in composite primary key scenarios without re-querying the computation node.

[0070] The primary key field identified based on constraint attribute data is used to filter primary key constraint-related entries from the constraint attribute data entries and extract the set of participating fields. The identification process includes scanning constraint attribute data entries, determining the primary key constraint identifier in the entries, and merging the set of participating fields that meet the conditions into the primary key field identification result. The primary key field identification result can semantically be a single-field set or a multi-field set; multi-field sets correspond to composite primary keys, and single-field sets correspond to non-composite primary keys. The identification result must maintain a consistent representation of field names. Field names can use the original field name or a standardized name. Standardized names are used to handle differences in capitalization, quotation mark enclosed names, or namespace prefixes, avoiding inconsistencies in field names during subsequent condition construction or parameter passing.

[0071] Determining whether the identified primary key field is a composite primary key containing multiple fields is used to determine the cardinality of the primary key field set and select different processing branches accordingly. The determination process includes obtaining the number of elements in the primary key field set and comparing the number of elements with a threshold (set to one) to distinguish between single-field and multi-field primary keys. The definition of a composite primary key describes that the primary key field set contains multiple fields that together constitute a unique identifier. Branch one corresponds to a composite primary key field, where the first field in the composite primary key is selected as the final primary key field according to the field definition order. The field definition order comes from the field definition set in the table structure definition information. The selection process includes reading the field definition order for each field of the composite primary key, sorting the field definition order, selecting the field with the smallest order, and writing the selection result into the final primary key field. Branch two corresponds to a primary key field that is not a composite primary key, where the identified primary key field is used as the final primary key field. The processing includes copying the field names from the single-field set to the final primary key field and performing type consistency processing. Type consistency processing is used to convert the set representation to a single-field representation to avoid differences between the set and single-value representations in subsequent use.

[0072] The finalized primary key field is used as the statistical field to determine and bind the statistical basis fields. The statistical field serves as the input field identifier for subsequent statistical processes and must be referable and verifiable. The binding process includes writing the finalized primary key field into the statistical field variable, storing the relationship between the statistical field and the target data table locally, and performing a consistency check on the existence of the statistical field in the field definition set. The consistency check includes field name matching and field comparability judgment. The field comparability judgment is used to confirm that the field type meets the requirements of statistical operations such as interval partitioning or boundary extraction. The field type can be a numeric type, a sortable string type, or a type that can be mapped to a sortable key, avoiding the selection of fields that cannot participate in interval partitioning, which would prevent subsequent statistical information from forming valid boundaries.

[0073] This embodiment sends metadata query requests and receives table structure definition information through a communication connection. It then parses constraint attribute data from the table structure definition information to identify primary key fields. In both composite primary key and non-composite primary key cases, it generates the final primary key fields and binds them as statistical fields. This makes the source of statistical field determination traceable and the processing branches complete, reducing inconsistencies in field selection and instability in subsequent statistical basis caused by differences in primary key form, and improving the repeatability and consistency of the statistical field determination process.

[0074] In one embodiment, step S40 above includes: S401, fill the statistical fields and statistical parameters into the preset instruction template to generate a statistical creation instruction; S402, the statistics creation command is sent to the computing node via a communication connection; S403, the computing node parses the statistics creation instruction to determine the task scheduling logic for concurrent distribution; S404, the computing node concurrently sends the statistical creation instruction to each storage node according to the distribution metadata information of the target data table and the task scheduling logic; S405, the data distribution analysis process for the statistical field is initiated by each of the storage nodes based on the statistical creation instruction; S406, Each of the storage nodes generates data distribution statistics based on the statistical parameters and the data distribution analysis process.

[0075] In this embodiment, the instruction template is used to constrain the structure and field set of statistical creation instructions. The instruction template includes a field carrying area, a parameter carrying area, and a control information area. The field carrying area is used to write the field names of the statistical fields. The field names can use the original field names of the target data table or standardized names. Standardized names are used to eliminate differences in case sensitivity, qualifier differences, and escape sequences. The parameter carrying area is used to write the parameter values ​​of the statistical parameters. The parameter values ​​can include parameters such as the number of buckets, interval granularity, and boundary generation strategy. Parameter items are encoded in the form of key-value pairs for easy parsing. The control information area is used to write the instruction type identifier, target object identifier, version identifier, and idempotency identifier. The idempotency identifier is used to prevent the same round of statistical creation from being triggered repeatedly when the same statistical creation instruction is delivered repeatedly. The process of filling statistical fields and parameters into the instruction template includes field validation, parameter validation, and serialization encoding. Field validation checks the existence and comparability of statistical fields in the target data table field definition set. Parameter validation checks that statistical parameters meet the value range and combination constraints. Serialization encoding converts the filled structure into a transmittable byte sequence or text sequence, and can attach a checksum for transmission integrity verification.

[0076] The communication connection is used to transmit and confirm statistics creation commands between the initiating end and the computing node. The communication connection includes a connection identifier, session state, and send / receive buffers. The sending process includes writing to the send buffer, triggering the send operation, waiting for the computing node's acknowledgment, and recording the acknowledgment status. The acknowledgment status is used to determine whether the statistics creation command has been received by the computing node and entered the parsing process. To reduce the impact of transmission anomalies on subsequent processes, the communication connection can be configured with timeout parameters and retransmission policies. The retransmission policy uses an idempotent flag for deduplication control, avoiding duplicate concurrent transmissions caused by retransmissions.

[0077] The compute node parsing and statistical creation instruction is used to extract the field-carrying area, parameter-carrying area, and control information area from the instruction template. The parsing result forms an executable scheduling input. The parsing process includes decoding, structural verification, and semantic verification. Decoding restores the byte sequence to a structured object. Structural verification verifies the integrity of the field set and version compatibility. Semantic verification verifies the executability of the combination of statistical fields and statistical parameters. The task scheduling logic describes the organization and resource allocation of concurrent delivery. The task scheduling logic may include a concurrency limit, delivery batch division rules, target storage node selection rules, and failure retry rules. The concurrency limit limits the number of target storage nodes that are simultaneously delivering data. Batch division rules divide the target storage node set into multiple batches to balance the sending pressure of compute nodes. Failure retry rules trigger retransmission and record the number of retries and backoff intervals when a storage node fails to acknowledge receipt or execution times out.

[0078] Distribution metadata describes the distribution of target data tables across storage nodes. This metadata includes mappings from shard identifiers to storage node identifiers, mappings from partition ranges to storage node identifiers, and descriptions of replica sets. Compute nodes use this distribution metadata to determine the target set for statistical creation instructions and generate a concurrent delivery plan based on task scheduling logic. The concurrent delivery plan includes a list of target storage node identifiers, instruction replicas for each target storage node, and delivery timing control information. The concurrent delivery execution process includes constructing a delivery message for each target storage node, attaching the target storage node identifier and idempotency identifier to the delivery message, sending the message in parallel, and collecting receipts. Receipts are used to mark whether a target storage node has entered the data distribution analysis process and also to trigger failure retry rules.

[0079] The storage node initiates a data distribution analysis process based on statistical creation instructions to generate statistical inputs and outputs around statistical fields within the local data storage scope. The data distribution analysis process includes data reading, field value extraction, boundary candidate generation, and statistical result organization. Data reading is performed locally on the storage node, and the reading scope is limited by the shard or partition range of the target data table on that storage node. Field value extraction extracts the field values ​​corresponding to the statistical fields from the read records and performs type normalization. Type normalization ensures consistency in comparison operations during subsequent boundary generation. Boundary candidate generation determines the number of buckets and the bucket boundary generation strategy based on statistical parameters. The bucket boundary generation strategy can adopt an equal-width strategy, an equal-frequency strategy, or a sampling-based approximation strategy, with the sampling scale and sampling method constrained by statistical parameters. Statistical result organization encapsulates bucket boundaries, bucket counts, and necessary context identifiers into intra-node fragments of data distribution statistics. These intra-node fragments can include the target data table identifier, statistical field identifier, statistical parameter summary, and bucket boundary value set, facilitating subsequent summarization and parsing.

[0080] The storage node generates data distribution statistics based on statistical parameters during the data distribution analysis process. This data distribution statistics are used to implement the constraints of the statistical parameters into the structure and granularity of the statistical output. The constraint on the number of buckets in the statistical parameters affects the length of the bucket boundary value set; the granularity constraint affects the selection of the interval or quantile point of the boundary values; and the truncation or overflow handling rules affect the closure method of the minimum and maximum boundaries. The generated data distribution statistics form a returnable result object on the storage node side. This result object includes a bucket boundary value set and a bucket count set. The bucket count set describes the distribution of the number of records corresponding to each bucket, while the bucket boundary value set describes the basis for dividing the value distribution range.

[0081] This embodiment uses an instruction template to structurally constrain the field and parameter carrying areas of the statistical creation instruction. The statistical creation instruction is then sent to the computing node via a communication connection. The computing node then combines the distributed metadata information and task scheduling logic to concurrently send the instruction to the storage node. This prompts the storage node to initiate the data distribution analysis process locally based on the statistical fields and parameters and generate data distribution statistics. This ensures that the input elements of statistical creation have consistent encoding and verification paths, and that the sending objects and concurrent organizations have controllable scheduling criteria. This reduces the deviations introduced by inconsistent parameters, inconsistent target selection, or inconsistent sending rhythms in a multi-node environment.

[0082] In one embodiment, step S50 above includes: S501, Receive the summary statistical result data returned by the computing node through the communication connection. The summary statistical result data is formed by summarizing the data distribution statistical information generated by each storage node and sending it back to the computing node. S502, perform a parsing operation on the summarized statistical results data to identify the data distribution statistics information corresponding to each of the storage nodes; S503, extract the bucket boundary values ​​based on the statistical fields from the data distribution statistics; S504, logically combine the bucket boundary values ​​to generate multiple data intervals describing the start and end points of the data record value distribution range on each storage node.

[0083] In this embodiment, the computing node acts as the receiver of external requests. After generating data distribution statistics on the storage node side, it is responsible for aggregating multiple sets of data distribution statistics into a summary statistical result and returning it via the communication connection. The summary statistical result is a transmittable aggregation object, which includes at least storage node identification information, target data table identification information, statistical field identification information, statistical parameter summary information, bucket boundary value set, bucket count set, and integrity verification information. The storage node identification information is used to distinguish statistical result fragments from different storage nodes; the target data table identification information is used to limit the data object corresponding to the statistical result; the statistical field identification information is used to constrain the comparison and sorting domains of the bucket boundary values; the statistical parameter summary information is used to record the number of buckets, boundary strategies, and sampling constraints used when generating the bucket boundary values; and the integrity verification information is used to identify abnormal situations such as transmission truncation, out-of-order splicing, and duplicate fragments. The communication connection is used to carry the return of the summary statistical results data. At the implementation level, the communication connection can be a long connection or a short connection. Long connections focus on maintaining stable session state and continuously pushing results, while short connections focus on one-time result return and controllable implementation of failure retries. When returning, fragmented transmission and streaming transmission can be used to adapt to large-scale bucket boundary value sets.

[0084] Upon receiving the aggregated statistical results data returned by the computing nodes via communication connections, the transmitted payload needs to be restored to a structured object in memory, and a consistency check needs to be performed on the aggregated object. The consistency check covers the uniqueness of storage node identification information, the matching of statistical field identification information, the type consistency of the bucket boundary value set, and the monotonicity and range closure of the bucket boundary value set. The uniqueness of storage node identification information prevents the same storage node result fragment from repeatedly entering subsequent parsing processes; the matching of statistical field identification information prevents the mixing of bucket boundary values ​​from different statistical fields; type consistency prevents comparison errors caused by mixing integer and string boundaries; monotonicity ensures that the bucket boundary value set satisfies increasing or decreasing constraints; and range closure ensures that the bucket boundary value set covers the minimum and maximum boundaries or corresponding open and closed interval rules agreed upon by the bucketing strategy. The aggregated statistical results data may include version identification information, which is used to select different field layouts and encoding methods during the parsing phase to prevent misparsing caused by field offsets after upgrades.

[0085] The parsing operation performs parsing operations on the summarized statistical results data and identifies the data distribution statistics corresponding to each storage node. The parsing operation includes four stages: splitting, locating, deserializing, and reassembling. Splitting divides the summarized statistical results data into multiple result segments according to the storage node identifier information. Locating locates the start and end offsets of the bucket boundary value sets and bucket count sets within each result segment. Deserializing restores the binary or text-encoded bucket boundary value sets to a comparable numerical sequence. Reassembling assembles the bucket boundary value sets and bucket count sets into a data distribution statistics information object and attaches the storage node identifier information. To improve parsing stability, the parsing operation can introduce dual constraints of a length prefix and a separator. The length prefix is ​​used to quickly locate the byte range of the boundary sets, and the separator is used to provide the minimum recoverable split point in case of abnormal truncation. The parsing operation can maintain an index table, which records the mapping relationship from storage node identifier information to the data distribution statistics information object. This mapping relationship is used for fast access in subsequent bucket boundary value extraction and interval combination stages.

[0086] Extracting bucket boundary values ​​based on statistical fields from data distribution statistics requires satisfying three constraints simultaneously: field consistency, sorting consistency, and deduplication consistency. Field consistency is achieved by comparing the statistical field identifiers in the data distribution statistics, ensuring the extracted object matches the statistical field. Sorting consistency is achieved by performing a stable sort on the bucket boundary value set or verifying its monotonicity. The sorting is based on the data type rules from the statistical fields: integers and floating-point numbers are compared numerically, time numbers are compared using timestamps, and strings are sorted lexicographically or according to a preset sorting rule. Deduplication consistency handles duplicate boundary values ​​between adjacent buckets and duplicate boundary values ​​across storage nodes. Deduplication strategies can include retaining a single boundary value and merging the corresponding bucket count, or retaining the boundary value and selecting an opening / closing rule during the interval combination stage to avoid interval overlap. Bucket boundary values ​​may include a minimum boundary sentinel and a maximum boundary sentinel. Sentinels are used to express the concepts of negative infinity or positive infinity, or to express the extended range agreed upon by statistical parameters. During the extraction stage, sentinels need to be mapped to computable boundary markers to participate in subsequent logical combinations.

[0087] The data intervals are generated by logically combining the bucket boundary values. This logical combination revolves around the rules for determining the start and end points. Each data interval consists of a start point and an end point. The start point originates from a boundary value in the bucket boundary value set, and the end point originates from an adjacent boundary value or a closed boundary generated by the rules. Logical combination requires defining interval types and comparison symbols. Interval types include four categories: left-closed and right-open, left-open and right-closed, double-closed, and double-open. The selection is based on interval closure rules from the statistical parameter summary information or data type rules from the statistical fields. Left-closed and right-open reduces the risk of duplicate hits due to overlapping adjacent interval boundaries, while double-closed is suitable for scenarios with discrete values ​​and indivisible boundary values. Logical combination needs to handle interval concatenation across storage nodes. The bucket boundary value sets generated by each storage node may have different granularities or coverage ranges. The combination stage can either generate data intervals independently for each storage node and output a data interval set with storage node identification information, or globally merge bucket boundary values ​​based on statistical fields and then generate a global data interval set. The collection of data intervals generated independently for each storage node helps maintain consistency between the distribution range of data records and the local data distribution statistics of the storage node. The global merging method facilitates the unification of interval granularity in subsequent stages. The merging phase also needs to handle empty bucket intervals and abnormal bucket intervals. Empty bucket intervals refer to data intervals corresponding to a bucket count of zero, while abnormal bucket intervals refer to intervals where the bucket boundary values ​​are reversed, missing, or cross abnormal ranges. Handling methods include removing empty bucket intervals, merging empty bucket intervals, or retaining empty bucket intervals and marking them as empty. The handling method is affected by the fault tolerance strategy constructed by subsequent export statements. When multiple data intervals are formed, interval index information can be generated synchronously. The interval index information records the sequence number of the data interval within the storage node and its global sequence number. The sequence number information is used to stably reproduce the interval order and support the consistency of subsequent task allocation.

[0088] This embodiment receives and parses the summarized statistical results data via a communication connection. It can split and identify the data distribution statistics information returned by the storage node according to the storage node identification information, and extract the bucket boundary values ​​under the consistency constraint of statistical fields. Under the condition that the bucket boundary values ​​meet the sorting consistency and deduplication consistency, the bucket boundary values ​​are logically combined into multiple data intervals containing start and end points according to the interval closure rule. This makes the data record value distribution range fall into multiple data interval sets in a calculable and verifiable form, thereby providing a clear boundary basis for subsequent expression and processing based on data intervals.

[0089] In one embodiment, step S60 above includes: S601, Obtain a preset export query template for the target data table, and identify the logical location in the export query template used to embed filter conditions; S602, traverse the multiple data intervals and parse out the corresponding start boundary value and end boundary value for each data interval; S603, Based on the statistical field, the starting boundary value, and the ending boundary value, construct interval filtering conditions for limiting the data reading range according to a preset comparison operator; S604, fill the interval filtering conditions into the logical position of the export query template to instantiate and generate the corresponding export statement for each data interval.

[0090] In this embodiment, multiple export statements are constructed based on data ranges, involving the combination and consistency constraints of seven types of elements: template objects, insertion position objects, range traversal objects, boundary value objects, comparison operator objects, range filtering condition objects, and export statement objects. The target data table is the object of the export statements. The identifier information of the target data table needs to be consistent with the table name string obtained from the export request parsing. The field set of the target data table needs to be consistent with and usable statistical fields to avoid the range filtering conditions referencing non-existent fields. The export query template is a reusable statement skeleton. It contains at least an access fragment, a result projection fragment, and a filter condition containment fragment for the target data table. The access fragment is used to determine the data source, the result projection fragment is used to determine the output column set, and the filter condition containment fragment is used to carry the range filtering conditions. The export query template can be obtained from a preset configuration, template library retrieval, or runtime assembly. The preset configuration can use text templates or structured templates. Template library retrieval can establish a mapping relationship based on the table name string of the target data table. Runtime assembly can fill the table name string of the target data table into a preset statement fragment to form the export query template. Exported query templates need to maintain syntactic closure. Replaceable placeholder structures should be reserved in the template to avoid disrupting the statement structure after the range filtering conditions are filled.

[0091] Identify the logical locations within the exported query template used to embed filtering conditions. These logical locations are internal template positioning information and can be represented as placeholder markers, syntax node positions, or character offset ranges. Placeholder markers explicitly indicate the insertion point in the template text; syntax node positions mark the WHERE or FILTER node in the template's abstract syntax tree; and character offset ranges mark the start and end positions of the inserted paragraph in the template string. Logical location identification needs to be compatible with both cases where there are existing and non-existent filtering conditions in the exported query template. When there are existing filtering conditions, the logical location is used to append range filtering conditions and handle AND connectors; when there are no filtering conditions, the logical location is used to generate the WHERE clause and embed range filtering conditions. Logical location identification also needs to be compatible with differences in the layout of filtering expressions across different dialects, such as placing the filtering expression in the WHERE clause or within a subquery. The logical locations are output as a unified positioning result during the template parsing phase for use in the subsequent population phase.

[0092] The process iterates through multiple data intervals, parsing the start and end boundary values ​​for each interval. These intervals belong to a set of objects to be transformed, and the iteration order and granularity must be clearly defined. The iteration order can be the natural order in which the data intervals were generated, the order sorted by the start and end boundary values, or the order sorted by the end boundary values. The chosen order affects the generation order of the exported statements but does not change the one-to-one correspondence between each exported statement and the data interval. For each data interval, the start and end boundary values ​​are parsed. This parsing process requires extracting boundary fields from the data interval structure and performing type restoration. Type restoration restores the start and end boundary values ​​to a data type consistent with the statistical fields, avoiding range offsets caused by string comparisons of numerical boundaries and time zone deviations caused by localized strings for time-related boundaries. The parsing process handles two types of boundary anomalies: missing boundaries and boundary sentinels. For missing boundaries, the minimum or maximum boundary can be filled in according to statistical parameter conventions. For boundary sentinels, they can be mapped to an expressible form of open or closed intervals, ensuring that the interval filtering conditions can be recognized by the database execution engine.

[0093] Interval filtering conditions are constructed based on statistical fields, starting boundary values, and ending boundary values, according to preset comparison operators. The statistical fields serve as the anchor points for the interval filtering conditions, while the starting and ending boundary values ​​define the value boundaries of the interval filtering conditions. The preset comparison operators define the opening and closing relationships and boundary inclusion relationships of the intervals. These preset comparison operators can be combinations of greater than or equal to, less than, greater than, and less than or equal to. The combination method must be consistent with the data interval generation rules to avoid overlap or gaps between adjacent data intervals at boundary points. The construction of interval filtering conditions requires the simultaneous generation of field expressions, boundary expressions, and logical connection expressions. Field expressions reference statistical fields, boundary expressions reference starting and ending boundary values, and logical connection expressions combine the constraints at both ends into a single filtering expression. The construction of boundary expressions requires handling the escaping and formatting of boundary values. Numerical boundaries can be expressed using direct literals, string boundaries need to be escaped according to database string rules, and time boundaries need to be expressed according to database time literal rules. In the interval filtering condition construction stage, a parameterization mechanism can be introduced to map the start and end boundary values ​​as parameter placeholders and generate a parameter binding set at the same time. This reduces the parsing and escaping differences introduced by concatenated statements. The parameterized interval filtering conditions still need to maintain the same syntax structure as the exported query template.

[0094] The process of filling range filtering conditions into the logical locations of the exported query template is used to instantiate and generate exported statements. During the filling phase, the range filtering conditions must be written into the logical locations and the statements closed. The exported query template and the logical locations jointly determine the filling method. When placeholders are used, the range filtering conditions replace the placeholder text using a substitution method. When syntax node locations are used, the range filtering conditions are attached to the filter nodes using a syntax tree insertion method. When character offset ranges are used, the range filtering conditions are inserted into the specified offset range using a concatenation method. The filling phase requires handling connector and parenthesis rules. If the exported query template already contains a filter expression, an AND statement needs to be inserted between the original filter expression and the range filtering conditions, and parentheses need to be added to the range filtering conditions to maintain consistent priority. If the exported query template does not contain a filter expression, a WHERE clause needs to be generated, with the range filtering conditions as the clause body. When instantiating and generating exported statements, a consistent mapping relationship between each data range and its corresponding exported statement must be maintained. This mapping relationship can be achieved by maintaining a set of exported statements in memory and retaining the data range index information. The set of exported statements is used to aggregate multiple exported statements to support subsequent task allocation, and the data range index information is used for task deduplication and retry positioning during the scheduling phase. After the export statement is generated, syntax validation and field validation need to be performed. Syntax validation is used to check the closure of brackets and the structure of keywords, while field validation is used to check the existence and comparability of statistical fields in the target data table. If the validation fails, the export query template or logical position identification result can be rolled back to avoid generating an unexecutable export statement.

[0095] This embodiment obtains the export query template and identifies the logical position, enabling stable insertion of interval filtering conditions. By traversing multiple data intervals and parsing the start and end boundary values, each data interval forms an expressible boundary pair. Interval filtering conditions are constructed using statistical fields, start and end boundary values, and preset comparison operators, allowing the data reading range to fall into the statement structure as a computable filtering expression. By filling the interval filtering conditions into the logical position of the export query template and instantiating and generating export statements, multiple export statements form a one-to-one correspondence with multiple data intervals and maintain executable syntactic closure. This transforms interval-level data reading requirements into a set of deployable statements and reduces the risk of interval overlap and omission.

[0096] In one embodiment, step S70 above includes: S701, Obtain the preset parallelism configuration value, and establish a communication link pool containing multiple concurrent connections with the computing node; S702, the exported statements are summarized into the set of tasks to be processed; S703, monitor the busy status of each concurrent connection in the communication link pool, the capacity of the communication link pool is determined by the parallelism configuration value, and generate a random index value to extract the exported statement non-sequentially from the set of tasks to be processed. S704, the extracted export statement is sent to the computing node through a non-busy concurrent connection, and the computing node determines the physical storage location according to the interval filtering conditions carried by the export statement. S705, the computing node transmits the exported statement to the corresponding storage node determined according to the physical storage location, so that each storage node can independently start data retrieval in the local storage layer.

[0097] In this embodiment, concurrent connections are used to carry the parallel delivery and execution feedback channels for exported statements. The establishment of concurrent connections requires a parallelism configuration value as an upper limit constraint on the number of connections. The parallelism configuration value is a resource control parameter, which can be derived from the parallelism field carried in the export request, the parallelism entry in the runtime parameter configuration file, or the parallelism policy value injected by the runtime environment. The parallelism configuration value participates in the connection number calculation during the connection creation phase and in the capacity consistency verification during the connection monitoring phase. When establishing a communication link pool containing multiple concurrent connections with the compute node, the communication link pool represents an aggregate container for concurrent connections. The communication link pool needs to maintain runtime information such as connection identifier, connection status, connection timeout parameters, sending window, and receiving window. The establishment process requires handshaking and authentication reuse for each concurrent connection. After a successful handshake, the concurrent connection is registered in the communication link pool and the busy state is initialized to a non-busy state. Concurrent connections that fail authentication are removed from the communication link pool and a re-establishment operation is triggered until the capacity corresponding to the parallelism configuration value is reached.

[0098] When exported statements are aggregated into the task set, the task set serves to hold the exported statements to be assigned and provide a searchable task view. The task set can be implemented using a queue structure, an array structure, or a hash index structure. A queue structure is suitable for sequential popping, an array structure is suitable for random index access, and a hash index structure is suitable for deduplication and status tracking. Before an exported statement enters the task set, it needs to undergo statement integrity verification. Integrity verification includes target data table reference verification, range filtering condition syntax fragment verification, and statistical field reference verification. Exported statements that pass verification are written to the task set and task identification information is recorded. Task identification information can consist of a data range index, an exported statement summary value, or an increasing sequence, used for subsequent retries and result aggregation.

[0099] Monitoring the busy status of each concurrent connection in the communication link pool is used to determine the set of available concurrent connections. The busy status needs to be updated jointly by the sending and receiving sides. The sending side marks a concurrent connection as busy while it is occupying its sending window, waiting for confirmation of reception from the compute node, or waiting for the storage node to return exported data. The receiving side marks the concurrent connection as not busy and updates its connection health information when it receives a completion flag, timeout event, or error code forwarded by the compute node. When the capacity of the communication link pool is determined by the parallelism configuration value, capacity control needs to simultaneously constrain the number of concurrent connections and the maximum number of tasks in transit for each concurrent connection. The number of tasks in transit can be achieved through the sending window size of each concurrent connection, thereby preventing a single concurrent connection from being overloaded with tasks, causing other concurrent connections to become idle.

[0100] Random index values ​​are generated and derived statements are extracted non-sequentially from the set of tasks to be processed for random allocation. The random index values ​​are task selection parameters. The generation of random index values ​​requires the current length of the set of tasks to be processed as the upper bound of the value range, and a skip rule is applied to completed or extracted task positions. Random index values ​​can be output by a pseudo-random number generator or by a hash function based on a combination of timestamps and connection identifiers. The pseudo-random number generator needs to maintain a seed to avoid short-cycle repetition, and the hash function needs to introduce a perturbation term to avoid generating a fixed sequence when the length of the set of tasks to be processed remains constant. The extraction action needs to map the random index values ​​to the target positions in the set of tasks to be processed and extract the corresponding derived statements. After extraction, the task status is marked as allocated in the set of tasks to be processed. The allocated status needs to be associated with concurrent connection identifiers to support reclamation and reassignment during retries.

[0101] When sending the extracted export statement to the compute node via a concurrent connection in a non-busy state, it is necessary to first select a set of available concurrent connections and execute connection selection rules. The connection selection rules can either perform a random selection from the non-busy set or prioritize connection health, which can be calculated from the most recent round-trip latency, error rate, and number of timeouts. Sending the export statement requires encapsulating the export statement and session identifier together into a request message. The request message can include task identifier information and a summary value of the interval filtering conditions, facilitating fast routing by the compute node during the parsing phase. When the compute node determines the physical storage location based on the interval filtering conditions carried in the export statement, the physical storage location represents the data sharding location information corresponding to the interval filtering conditions. The physical storage location can be represented as a shard identifier, partition identifier, or a set of storage node identifiers. The determination process requires performing sharding rule matching on the statistical fields, start boundary values, and end boundary values ​​in the interval filtering conditions. The sharding rules can come from the distribution metadata information of the target data table. The matching result outputs the physical storage location and is associated with the task identifier information.

[0102] When a compute node forwards an exported statement to the corresponding storage node determined by its physical storage location, it needs to map the physical storage location to the storage node's network address or storage node identifier, and trigger data retrieval execution on the storage node side. The forwarding action maintains the semantics of the exported statement and retains the task identification information. After receiving the exported statement, when a storage node independently initiates data retrieval in its local storage layer, it needs to submit the exported statement to the local executor and bind the index access path or scan access path corresponding to the range filtering conditions. Independently initiating data retrieval means that each storage node's execution is independent of others; storage nodes do not share execution context. The execution status of a storage node is returned to the compute node using the task identification information as an index, or directly returned to the receiving side of the concurrent connection, thus completing the concurrent scheduling and execution state closure.

[0103] For example, the export tool receives an export request for a target data table. This request carries export control information such as the table name, the set of fields to be exported, and row / column separator configurations. The export tool performs syntax parsing on the export request to obtain the table name string of the target data table. Based on this string, it queries the node mapping relationship to determine the network address and service port number of the compute node. The export tool calls the database connection driver to initiate a network connection handshake with the compute node based on the network address and service port number, submitting access authentication credentials. After the compute node returns an authentication success signal, the export tool confirms the establishment of a communication connection with the compute node. Once the communication connection is established, the export tool sends a metadata query request to the compute node through the connection. The compute node returns the table structure definition information of the target data table. The export tool parses constraint attribute data from the table structure definition information and identifies the primary key field with a primary key constraint identifier based on the constraint attribute data. The export tool determines whether the primary key field is a composite primary key containing multiple fields. If the primary key field is a composite primary key, the first field in the composite primary key is selected as the final primary key field according to the field definition order. If the primary key field is not a composite primary key, the identified primary key field is used as the final primary key field. The export tool then uses the final primary key field as the statistical field.

[0104] After determining the statistical fields, the export tool identifies the statistical parameters used to create data distribution statistics. These parameters define the granularity and range of the data distribution statistics. The statistical parameters can be calculated based on the number of records in the target data table across different storage nodes and the expected number of records per interval. The calculated number of buckets is then written into the statistical parameters to form a parameter structure that can be parsed by the compute nodes. The export tool fills the statistical fields and parameters into a preset instruction template to generate a statistical creation instruction, which is then sent to the compute nodes via a communication connection. The compute nodes parse the statistical creation instruction and determine the task scheduling logic for concurrent distribution. Combining the distribution metadata of the target data table with the task scheduling logic, the compute nodes concurrently distribute the statistical creation instruction to each storage node. Each storage node initiates the data distribution analysis process for the statistical fields based on the statistical creation instruction and generates data distribution statistics according to the statistical parameters. The data distribution statistics describe the segment boundaries of the statistical fields in the form of bucket boundary values. Each storage node sends the generated data distribution statistics back to the compute node. The compute node summarizes the data to form a summary statistical result and returns it to the export tool via a communication connection. The export tool performs a parsing operation on the summary statistical result to identify the data distribution statistics corresponding to each storage node. Then, it extracts the bucket boundary values ​​based on the statistical fields from the data distribution statistics and logically combines the bucket boundary values ​​to generate multiple data intervals that describe the start and end points of the data record value distribution range on each storage node.

[0105] After multiple data ranges are generated, the export tool obtains the preset export query template for the target data table and identifies the logical locations within the template used to insert filtering conditions. The export tool iterates through the multiple data ranges and parses out the corresponding start and end boundary values ​​for each range. Based on the statistical field, start and end boundary values, the export tool constructs range filtering conditions according to preset comparison operators and fills these conditions into the logical locations of the export query template to instantiate and generate the corresponding export statement. For example, when the statistical field is the integer field `id`, the data ranges generated by a certain storage node could be (1, 50000), (50001, 100000), or (100001, 150000). Based on this, the export tool constructs the corresponding range filtering condition for the export statement as `where id>= 1 and id`. <50000、where id> =50001 and id <100000、where id> = 100001 and id<= 150000. The data range generated by another storage node can be (1, 30000), (50000, 80000), or (90000, 120000). Based on this, the export tool constructs the corresponding range filtering condition for the export statement as WHERE id>= 1 AND id <30000、where id> = 50000 and id <80000、whereid> = 90000 and id<= 120000. Each export statement corresponds one-to-one with a data range, and each export statement contains the range filtering conditions for the corresponding data range, thus ensuring that the search scope of the export statement is consistent with the data range.

[0106] After generating the export statement, the export tool obtains the preset parallelism configuration value and establishes multiple concurrent connections with the computing nodes based on the parallelism configuration value to form a concurrent connection pool. This pool serves as the concurrency channel for the parallel delivery and execution feedback of the export statement. The export tool aggregates the export statements into a set of tasks to be processed and monitors the busy status of each concurrent connection in the pool. It generates a random index value within the index range of the task set and extracts export statements from the task set non-sequentially based on this random index value. The export tool sends the extracted export statements to the computing nodes through a non-busy concurrent connection. The computing nodes determine the physical storage location based on the range filtering conditions carried in the export statement and forward the export statement to the corresponding storage node determined by the physical storage location. The corresponding storage node independently initiates data retrieval in its local storage layer and returns the exported data. The computing nodes forward the exported data to the export tool, which receives and stores the exported data forwarded by the computing nodes through concurrent connections. The storage of exported data can be combined with the row and column separator configuration in the export request to perform record splicing and disk writing, so that the exported data received by different concurrent connections can be written to the target storage medium in a unified format and form an exported file with the output path specified in the export request.

[0107] This embodiment achieves controllable parallel resource boundaries for export statement distribution by configuring numerical constraints on the number of concurrent connections and establishing a communication link pool. It provides a manageable basis for task extraction, allocation, and retries by carrying export statements through a set of pending tasks and introducing task identification information. By monitoring the busy status of concurrent connections and sending export statements to concurrent connections in a non-busy state, it matches task allocation with connection availability and reduces idle waiting. By randomly extracting export statements from the set of pending tasks using random index values, it creates a discretized distribution of export statements among concurrent connections and reduces load skew caused by centralized allocation. Finally, by having computing nodes determine physical storage locations based on interval filtering conditions and pass-through export statements to the corresponding storage nodes, it enables data retrieval to be initiated in parallel on multiple storage nodes while maintaining interval-level routing consistency. This improves the parallelism between export statement distribution and storage node execution and reduces the waiting overhead caused by single-point serial scheduling.

[0108] In one embodiment, a concurrent export device based on data distribution statistics is provided, which corresponds one-to-one with the concurrent export method based on data distribution statistics in the above embodiments. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the concurrent export device based on data distribution statistics of the present invention. The modules include: request receiving and connection management module 10, primary key field extraction and determination module 20, statistical parameter generation module 30, statistical creation instruction sending module 40, data range acquisition and processing module 50, export statement construction module 60, concurrent connection management and task allocation module 70, and exported data receiving and storage module 80. Detailed descriptions of each functional module are as follows: The request receiving and connection management module 10 is used to receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database. The primary key field extraction and determination module 20 is used to obtain the primary key field of the target data table from the computing node and determine the primary key field as a statistical field; The statistical parameter generation module 30 is used to determine the statistical parameters used to create data distribution statistics. The statistics creation instruction sending module 40 is used to send statistics creation instructions to the computing node, and drive each storage node to create data distribution statistics information based on the statistics fields and statistics parameters. The data interval acquisition and processing module 50 is used to acquire multiple data intervals contained in the data distribution statistics information fed back by each of the storage nodes from the computing nodes; The export statement construction module 60 is used to construct multiple export statements based on the data range, and each export statement contains a range filtering condition corresponding to one of the data ranges. The concurrent connection management and task allocation module 70 is used to establish multiple concurrent connections with the computing node and randomly allocate the exported statement to the concurrent connections for distribution, so as to drive each of the storage nodes to concurrently execute data retrieval. The exported data receiving and storage module 80 is used to receive exported data forwarded by the computing node through the concurrent connection and to store the exported data.

[0109] Specific limitations regarding the concurrent export device based on data distribution statistics can be found in the aforementioned limitations on the concurrent export method based on data distribution statistics, and will not be repeated here. Each module in the aforementioned concurrent export device based on data distribution statistics can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0110] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides deterministic and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a concurrent derivation method based on data distribution statistics.

[0111] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a concurrent derivation method based on data distribution statistics.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database; Obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field; Determine the statistical parameters used to create data distribution statistics; Send a statistics creation instruction to the computing node, and drive each storage node to create data distribution statistics based on the statistics fields and parameters; The computing node obtains multiple data intervals contained in the data distribution statistics fed back by each of the storage nodes; Multiple export statements are constructed based on the data range, and each export statement contains a range filtering condition for a corresponding data range. Establish multiple concurrent connections with the computing node, and randomly distribute the exported statement to the concurrent connections to drive each of the storage nodes to concurrently execute data retrieval; The concurrent connection receives and stores the exported data forwarded by the computing node.

[0113] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database; Obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field; Determine the statistical parameters used to create data distribution statistics; Send a statistics creation instruction to the computing node, and drive each storage node to create data distribution statistics based on the statistics fields and parameters; The computing node obtains multiple data intervals contained in the data distribution statistics fed back by each of the storage nodes; Multiple export statements are constructed based on the data range, and each export statement contains a range filtering condition for a corresponding data range. Establish multiple concurrent connections with the computing node, and randomly distribute the exported statement to the concurrent connections to drive each of the storage nodes to concurrently execute data retrieval; The concurrent connection receives and stores the exported data forwarded by the computing node.

[0114] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0116] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A concurrent derivation method based on data distribution statistics, characterized in that, Includes the following steps: Receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database; Obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field; Determine the statistical parameters used to create data distribution statistics; Send a statistics creation instruction to the computing node, and drive each storage node to create data distribution statistics based on the statistics fields and parameters; The computing node obtains multiple data intervals contained in the data distribution statistics fed back by each of the storage nodes; Multiple export statements are constructed based on the data range, and each export statement contains a range filtering condition for a corresponding data range. Establish multiple concurrent connections with the computing node, and randomly distribute the exported statement to the concurrent connections to drive each of the storage nodes to concurrently execute data retrieval; The concurrent connection receives and stores the exported data forwarded by the computing node.

2. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, Receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database, including: Listen to the command data stream input from the user interaction interface and capture export requests containing the identification information of the target data table; The export request is parsed to extract the table name string of the target data table; Based on the table name string of the target data table, query the node mapping relationship to determine the network address and service port number of the corresponding computing node; The database connection driver is invoked to initiate a network connection handshake with the computing node based on the network address and the service port number, and access authentication credentials are submitted. After the computing node returns an authentication successful response signal, the communication connection with the computing node is confirmed to be established.

3. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, Obtain the primary key field of the target data table from the computing node, and determine the primary key field as a statistical field, including: Send a metadata query request to the computing node through the communication connection; Receive the table structure definition information of the target data table returned by the computing node; The constraint attribute data is parsed from the table structure definition information; Identify the primary key field with primary key constraint identifier based on the constraint attribute data; Determine whether the identified primary key field is a composite primary key containing multiple fields; If the primary key field is a composite primary key, then the first field in the composite primary key is selected as the final primary key field according to the field definition order; If the primary key field is not a composite primary key, then the identified primary key field will be used as the final determined primary key field; Use the final determined primary key field as the statistical field.

4. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, Sending a statistics creation command to the computing node, and based on the statistics fields and parameters, driving each storage node to create data distribution statistics information through the computing node, including: Fill the statistical fields and statistical parameters into the preset instruction template to generate a statistical creation instruction; The statistics creation command is sent to the computing node via a communication connection; The computing node parses the statistics creation instruction to determine the task scheduling logic for concurrent delivery; The computing node concurrently sends the statistical creation instruction to each storage node based on the distribution metadata information of the target data table and the task scheduling logic. Each of the aforementioned storage nodes initiates a data distribution analysis process for the statistical field based on the statistical creation instruction; Each of the aforementioned storage nodes generates data distribution statistics based on the statistical parameters and the data distribution analysis process.

5. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, The data distribution statistics returned by each of the storage nodes from the computing nodes include multiple data intervals, including: The system receives the summary statistical results data returned by the computing node through the communication connection. The summary statistical results data is generated by each storage node sending back the generated data distribution statistics information to the computing node and then summarizing the data. Perform a parsing operation on the summarized statistical results data to identify the data distribution statistics corresponding to each of the storage nodes; Extract the bucket boundary values ​​based on the statistical fields from the data distribution statistics; The bucket boundary values ​​are logically combined to generate multiple data intervals that describe the start and end points of the data record value distribution range on each storage node.

6. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, Multiple export statements are constructed based on the data range, and each export statement contains a range filtering condition for a corresponding data range, including: Obtain a preset export query template for the target data table, and identify the logical locations in the export query template used to embed filter conditions; Traverse the multiple data intervals and parse out the corresponding start boundary value and end boundary value for each data interval; Based on the statistical field, the starting boundary value, and the ending boundary value, an interval filtering condition for limiting the data reading range is constructed according to a preset comparison operator. The interval filtering conditions are filled into the logical positions of the export query template to instantiate and generate corresponding export statements for each data interval.

7. The concurrent export method based on data distribution statistics as described in claim 1, characterized in that, Establishing multiple concurrent connections with the computing nodes, and randomly distributing the exported statements to these concurrent connections to drive each storage node to concurrently execute data retrieval, includes: Obtain the preset parallelism configuration value and establish a communication link pool containing multiple concurrent connections with the computing node; The exported statements are aggregated into a set of tasks to be processed. Monitor the busy status of each concurrent connection in the communication link pool, the capacity of the communication link pool is determined by the parallelism configuration value, and generate random index values ​​to extract the exported statements non-sequentially from the set of tasks to be processed. The extracted export statements are sent to the computing node through a non-busy concurrent connection, and the computing node determines the physical storage location based on the range filtering conditions carried by the export statements. The computing node transmits the exported statement to the corresponding storage node determined according to the physical storage location, so that each storage node can independently start data retrieval in the local storage layer.

8. A concurrent export device based on data distribution statistics, characterized in that, The concurrent export device based on data distribution statistics includes: The request receiving and connection management module is used to receive export requests for the target data table and establish communication connections with the computing nodes of the distributed database. The primary key field extraction and determination module is used to obtain the primary key field of the target data table from the computing node and determine the primary key field as a statistical field; The statistical parameter generation module is used to determine the statistical parameters used to create data distribution statistics. The statistics creation instruction sending module is used to send statistics creation instructions to the computing node, and drive each storage node to create data distribution statistics information based on the statistics fields and statistics parameters. The data interval acquisition and processing module is used to acquire multiple data intervals contained in the data distribution statistics information fed back by each of the storage nodes from the computing nodes; The export statement construction module is used to construct multiple export statements based on the data range, and each export statement contains a range filtering condition for a corresponding data range. The concurrent connection management and task allocation module is used to establish multiple concurrent connections with the computing node and randomly allocate the exported statement to the concurrent connections for distribution, so as to drive each of the storage nodes to concurrently execute data retrieval. The exported data receiving and storage module is used to receive exported data forwarded by the computing node through the concurrent connection and to store the exported data.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a concurrent export program based on data distribution statistics stored in the memory and executable on the processor. When executed by the processor, the concurrent export program based on data distribution statistics implements the steps of the concurrent export method based on data distribution statistics as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a concurrent export program based on data distribution statistics, which, when executed by a processor, implements the steps of the concurrent export method based on data distribution statistics as described in any one of claims 1-7.

Images