Distributed data processing system based on big data mining
By establishing a two-way closed-loop feedback channel and a global cost model in the distributed data processing system, the problems of untimely index structure updates and resource waste are solved, lock-free smooth updates and efficient resource utilization are achieved, and system performance is improved.
Patent Information
- Application Number
- CN202611354719.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
When the distribution of data flow characteristics shifts in existing distributed data processing systems, the lack of a closed-loop feedback mechanism leads to untimely updates to the index structure, causing read-write lock conflicts and service interruptions. At the same time, the query optimizer fails to effectively utilize the storage-side hardware acceleration unit, resulting in a waste of network and computing resources.
A two-way closed-loop feedback channel is established between the mining computing engine and the underlying index structure. Layered asynchronous hot updates of the index are performed through drift metric and dynamic threshold determination. A global cost model is constructed to adaptively push up the cost operator to the storage-side hardware acceleration unit for execution.
It enables smooth updates to the index structure without interrupting online services, improving retrieval accuracy, alleviating network bottlenecks, and enhancing system throughput and resource utilization efficiency.
Smart Images

Figure CN123115244A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of distributed computing and big data mining technology, specifically a distributed data processing system based on big data mining. Background Technology
[0002] Modern distributed data processing platforms widely adopt a storage-compute separation architecture to support high-concurrency mining and analysis tasks of massive amounts of data. The storage layer is responsible for data persistence and index maintenance, while the compute layer is responsible for executing mining algorithms. Regarding distributed indexes, mainstream solutions are based on log-structured merged tree variant indexes or inverted indexes, with index weights typically determined statically based on the data distribution at the time of index creation. Furthermore, as the data flow characteristics change over time, existing systems lack a closed-loop mechanism to feed back upper-layer mining results to the lower-layer index structure. The closest existing technology to this invention employs an offline, periodic, full-scale index rebuilding combined with centralized filtering on the compute side: the system locks the target storage node at fixed intervals and rescans all historical data to generate a new index. After completion, it replaces the old index to restore read / write operations. During query execution... The current approach, which only performs partition pruning on storage nodes and then transmits the pruned data to compute nodes for centralized filtering and mining operations, suffers from several drawbacks. Firstly, index rebuilding locks storage nodes, which can lead to read / write lock conflicts and service interruptions in high-frequency write scenarios. Secondly, the fixed timeframe is ill-suited to adapting to dynamic changes in data distribution, resulting in decreased retrieval accuracy. Thirdly, the query optimizer lacks comprehensive consideration of network bandwidth, heterogeneous hardware computing power, and energy consumption. Fourthly, the large-scale data migration generated by long-tail queries can quickly consume network bandwidth, creating a distributed input / output bottleneck and limiting the overall system throughput. Finally, the programmable hardware acceleration units on the storage side are not effectively utilized by the query optimizer, resulting in wasted network and computing resources. Therefore, we propose a distributed data processing system based on big data mining. Summary of the Invention
[0003] The purpose of this invention is to provide a distributed data processing system based on big data mining.
[0004] The technical problem this invention aims to solve is that existing distributed data processing systems lack a closed-loop mechanism to feed back the drift perception results of the mining algorithm layer to the underlying index structure when the data flow feature distribution shifts. They can only rely on offline periodic full index reconstruction for updates. This method is prone to read-write lock conflicts and service interruptions in high-concurrency real-time write scenarios, and the fixed period is difficult to adapt to the dynamic changes in data distribution. At the same time, when processing long-tail queries, the query optimizer of existing systems lacks a comprehensive consideration of network bandwidth, heterogeneous hardware computing power and energy consumption costs, resulting in a large amount of raw data being moved to computing nodes for centralized processing, creating network input and output bottlenecks. Furthermore, it fails to effectively utilize the programmable hardware acceleration units on the storage side to perform data source filtering, resulting in a waste of network and computing resources.
[0005] To address the aforementioned technical problems, this invention provides a distributed data processing system based on big data mining. This system includes a control node, a computing node cluster, a storage node cluster, and a hardware acceleration unit deployed on the storage node cluster. Each storage node in the storage node cluster is equipped with an incremental data statistical probe, an index structure, and a multi-version concurrency control module. The incremental data statistical probe periodically collects the feature distribution of the data shards managed by the storage node according to a preset time window. The control node includes a drift evaluation module and a multi-dimensional cost optimizer: the drift evaluation module calculates the drift degree based on the collected feature distribution and historical baseline distribution, and compares the drift degree with a judgment threshold dynamically adjusted based on historical statistical information. When the drift degree exceeds the judgment threshold, an index reconstruction metadata plan is generated and sent to the corresponding target storage node. After receiving the plan, the target storage node uses the multi-version concurrency control module to refactor the current index. The structure creates a virtual snapshot, performs background asynchronous index reconstruction on the virtual snapshot, and after reconstruction, switches the read request routing pointer from pointing to the original index structure to pointing to the newly built index structure through an atomic pointer switching operation, thereby achieving lock-free smooth replacement. The multidimensional cost optimizer constructs a global cost model based on network transmission cost, computation cost, and energy consumption cost, and calculates the total cost of high-cost operators in the query execution path under the path of being pulled up to the centralized processing path of the computing node and the total cost under the path of being pushed down to the local processing path of the storage node. The difference between the two is used as the push-down benefit. When the push-down benefit is greater than zero, the control node compiles the high-cost operator into an instruction that can be executed on the hardware acceleration unit and sends it to the hardware acceleration unit of the target storage node. After the hardware acceleration unit performs operations such as filtering, regular expression matching, or feature extraction on the local data, it only sends the processed valid feature results back to the computing node that initiated the request through the network.
[0006] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:
[0007] 1. This invention establishes a bidirectional closed-loop feedback channel based on drift quantification and dynamic threshold determination between the mining computing engine layer and the underlying index structure, and combines virtual snapshot isolation reconstruction and atomic pointer switching mechanism, so that the index structure can achieve layered asynchronous hot updates without interrupting online read and write services and without generating read and write lock conflicts. This avoids the service interruption problem caused by traditional offline full reconstruction, and improves the retrieval accuracy of mining queries.
[0008] 2. This invention constructs a global cost model that integrates network transmission cost, computation cost, and energy consumption cost, and combines real-time telemetry data to adaptively determine the operator push-down benefit. This enables the system to accurately push high-cost operators down to the storage-side hardware acceleration unit for local execution, reducing the amount of invalid data transmitted over the network, alleviating the network input / output bottleneck problem caused by long-tail queries, and improving the overall throughput and resource utilization efficiency of the system. Attached Figure Description
[0009] Figure 1 This is a schematic diagram of the overall business process of the system in an embodiment of the present invention;
[0010] Figure 2 This is a schematic diagram of the drift-aware distributed index dynamic reconstruction process in an embodiment of the present invention;
[0011] Figure 3 This is a schematic diagram of the adaptive push-down scheduling process for multi-dimensional perception long-tail query in an embodiment of the present invention. Detailed Implementation
[0012] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.
[0013] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0014] I. System Overall Architecture:
[0015] The distributed data processing system based on big data mining provided by this invention includes a control node, a computing node cluster, a storage node cluster, and a hardware acceleration unit deployed on the storage node cluster. The hardware acceleration unit is a network hardware device with programmable data processing capabilities, which can directly execute data filtering, parsing and other computing tasks on the network transmission path.
[0016] The storage node cluster consists of several storage nodes. Each storage node is equipped with an incremental data statistics probe, an index structure, and a multi-version concurrency control module, and is connected to the local hardware acceleration unit. The incremental data statistics probe is used to periodically collect feature distribution samples of the data shards managed by the storage node according to a preset time window. The index structure can be a log structure merged tree variant index or an inverted index for text and semi-structured data. The multi-version concurrency control module is used to maintain multiple historical versions for the current index structure during index reconstruction, so as to achieve read and write concurrency without lock conflicts.
[0017] The computing node cluster includes several computing nodes, which are used to execute the main operation logic of the mining algorithm and provide mining service interfaces to the outside world. The computing node cluster also includes an upper-layer collaborative module of incremental data statistical probe, namely the mining algorithm execution module, which works with the drift evaluation function to form a feedback channel between the mining algorithm layer and the underlying index layer.
[0018] The control node is the core of this system for achieving closed-loop control. It includes two functional modules: a drift evaluation module and a multidimensional cost optimizer. The drift evaluation module is responsible for receiving the feature distribution acquisition results reported by each storage node, calculating the data distribution drift degree, and generating an index reconstruction metadata plan. The multidimensional cost optimizer is responsible for building a global cost model based on real-time telemetry data, calculating the push-down benefit of high-cost operators in long-tail queries, and generating push-down scheduling decisions. The control node is connected to the storage node cluster, the computing node cluster, and the hardware acceleration unit through a network. A real-time telemetry module based on eBPF technology is deployed on the network link to collect network bandwidth utilization, processor utilization, hardware acceleration unit utilization, and energy consumption indicators in real time for use by the multidimensional cost optimizer.
[0019] II. Drift-aware distributed index dynamic reconstruction mechanism:
[0020] The core idea of this mechanism is to establish a two-way closed-loop feedback channel between the mining computing engine and the storage layer index structure, and drive the hierarchical asynchronous hot update of the index by quantifying the data distribution drift, thereby avoiding the lock contention and service interruption problems caused by offline full reconstruction.
[0021] In terms of drift quantification, the drift evaluation module adopts... Divergence measures the difference between the current time window's feature distribution and the historical baseline distribution. The calculation formula is as follows:
[0022] ;
[0023] In the formula, This indicates the degree of drift of the current window feature distribution relative to the historical baseline distribution. This indicates that the feature value falls within the first time window. The probability of a discrete interval. This indicates that the eigenvalue in the historical baseline distribution falls into the th . The probability of a discrete interval. This represents the total number of bins for discretizing the eigenvalues;
[0024] To avoid using a fixed threshold to determine whether the drift is significant, the system uses a dynamic threshold based on historical statistics, calculated as follows:
[0025] ;
[0026] In the formula, Indicates the first The dynamic judgment threshold used in each monitoring cycle This represents the average drift calculated over multiple historical monitoring periods. This represents the standard deviation of the drift calculated over multiple historical monitoring periods. This represents the sensitivity coefficient, used to control the strictness of threshold judgment, and is preset by the system administrator based on business tolerance.
[0027] In the specific implementation process, the incremental data statistical probe deployed on each storage node of the distributed storage node cluster periodically collects the feature distribution samples of the data shards managed by the node according to the preset time window, and sends the collection results to the drift evaluation module of the mining computing engine. The drift evaluation module calculates the drift degree between the feature distribution of the current time window and the historical benchmark distribution according to the aforementioned KL divergence calculation formula, and updates the judgment threshold of the current period according to the aforementioned dynamic threshold calculation formula.
[0028] Subsequently, the drift assessment module determines whether the calculated drift degree is greater than the current dynamic threshold. If the result is no, the current index structure remains unchanged and the monitoring of the next cycle begins. If the result is yes, the control node receives the drift over-limit signal, generates an index reconstruction metadata plan, which includes the data shard identifier of the index to be reconstructed, the target storage node identifier, and the index parameters used for reconstruction, and sends the plan to the corresponding target storage node.
[0029] After receiving the index reconstruction metadata plan, the target storage node creates a virtual snapshot of the current index structure based on the multi-version concurrency control mechanism. This virtual snapshot is isolated from the original index structure accessed by online read and write requests at the storage level. The target storage node then performs background asynchronous index reconstruction operations on this virtual snapshot: if the index structure is a log structure merged tree variant, the hierarchical data blocks are reordered and merged according to the new merging strategy; if the index structure is an inverted index, the term weights are recalculated based on the current feature distribution and the inverted list is updated incrementally. The entire reconstruction process does not occupy lock resources for online read and write requests, so it does not affect the retrieval services provided by the original index structure.
[0030] After the background asynchronous index reconstruction operation is completed, the target storage node switches the read request routing pointer from pointing to the original index structure to pointing to the newly constructed index structure through an atomic pointer switching operation. This switching process is a single, indivisible atomic operation. Read requests arriving before and after the switch can receive consistent service without generating intermediate states, thus achieving seamless and smooth replacement without service interruption. After the switch is completed, the target storage node updates the reconstructed feature distribution as the new historical baseline distribution and releases the storage resources occupied by the original index structure, and then enters the next monitoring cycle.
[0031] III. Implementation details of MVCC virtual snapshots and atomic pointer switching:
[0032] To further illustrate how to avoid read-write lock conflicts during hot index updates, this invention sets up an index pointer table inside the storage node. This pointer table maintains the reference address of the index structure currently providing services to the outside world. During index reconstruction, the system generates a logically independent virtual snapshot based on the data foundation of the original index without affecting the external services of the original index structure. The background reconstruction process only performs write operations on this virtual snapshot. After reconstruction is completed, the system calls the underlying atomic operation instruction to replace the reference address recorded in the index pointer table with the address of the new index structure. This replacement operation is completed within a single processor instruction cycle. During this period, all read requests are either routed to the old index or to the new index, with no intermediate state, thus achieving seamless and smooth replacement. The entire process can be divided into three stages: the first stage is for the original index structure to provide services to the outside world, and the new version of the virtual snapshot is asynchronously reconstructed in the background; the second stage is for the virtual snapshot reconstruction to be completed, triggering atomic pointer switching; and the third stage is for the pointer field in the index pointer table to be changed to point to the new index structure, and the storage resources occupied by the original index structure are released.
[0033] IV. Multi-dimensional cost-aware adaptive pushdown scheduling mechanism for long-tail queries:
[0034] The core idea of this mechanism is to build a global cost model that integrates network transmission cost, computation cost and energy consumption cost at the query optimizer level, and adaptively determine whether to push down to the storage-side hardware for execution of high-cost operators in long-tail queries.
[0035] The system abstracts the total cost of the query execution path into a weighted sum of three parts: network transmission cost, computation cost, and energy consumption cost. The calculation formula is as follows:
[0036] ;
[0037] In the formula, This represents the total cost corresponding to a certain execution path. This indicates the estimated network transmission cost under this execution path, which is determined by both the amount of data to be transmitted and the current network bandwidth utilization. This indicates the estimated computational cost under this execution path, which is jointly determined by the operator's computational complexity and the utilization rate of the target node's processor or hardware acceleration unit. This represents the estimated energy consumption cost under this execution path. , , These are the weighting coefficients corresponding to the three costs: network, computation, and energy consumption. The three satisfy... Furthermore, it dynamically adjusts based on the node load status collected by the real-time telemetry module;
[0038] The system calculates the total cost of pulling data up to the centralized processing path of the computing nodes. The total cost of pushing the operator down to the local processing path of the storage node. The difference between the two is calculated as the downshortage benefit, and the calculation formula is as follows:
[0039] ;
[0040] In the formula, This represents the cost-benefit ratio of pushing the operator down to execute it versus pulling it up to execute it. When the value is greater than zero, it indicates that the total cost of pushdown execution is lower, and the pushdown scheme should be adopted. When the value is less than or equal to zero, it indicates that the total cost of centralized execution is lower, and the original execution path should be maintained.
[0041] In the specific implementation process, the control node receives the mining query request initiated by the computing node, performs syntax and semantic parsing on the query request, and generates an initial logical execution plan containing operators such as scanning, filtering, regular expression matching, and feature extraction. The control node collects the current network bandwidth utilization, processor utilization, hardware acceleration unit utilization, and energy consumption indicators of each physical node through real-time telemetry modules deployed on each storage node and network link, and summarizes the collected results into a weight matrix.
[0042] The multidimensional cost optimizer substitutes the weight matrix obtained from the aggregation into the aforementioned multidimensional cost model formula to calculate the total cost of high-cost operators in the initial logical execution plan under the pull-up centralized processing path and the total cost under the push-down to the storage node processing path. This calculation process solves the optimal combination of execution paths under given resource constraints using mixed integer linear programming.
[0043] After calculating the push-down benefit according to the aforementioned formula, the multidimensional cost optimizer determines whether the push-down benefit is greater than zero. If the result is no, the initial logical execution plan is maintained, and the control node instructs the storage node to pull up and transmit the pruned data to the computing node. The computing node then centrally executes the remaining filtering and mining calculations and generates query results. If the result is yes, the multidimensional cost optimizer compiles the high-cost operators that should be pushed down in the initial logical execution plan into bytecode or field-programmable gate array instructions that can run on the storage-side hardware acceleration unit through just-in-time compilation. The control node sends the compiled instructions to the hardware acceleration unit of the target storage node. Before the data is read from the storage medium and transmitted through the network interface, the hardware acceleration unit directly performs filtering, regular expression matching, or feature extraction operations on the local data. After that, the target storage node only sends the effective feature results obtained after processing by the hardware acceleration unit back to the requesting computing node through the network. The computing node then performs subsequent mining and aggregation operations on the returned feature results and generates query results.
[0044] To facilitate understanding, the following example is provided: Suppose a long-tail query involves a total of one thousand data units. If the pull-up centralized processing path is used, all one thousand data units need to be transmitted to the computing node via the network. The network transmission cost is proportional to the amount of data. If the push-down processing path is used, the hardware acceleration unit only needs to return a small number of valid feature results that meet the conditions after filtering on the storage side. At this time, the network transmission cost is greatly reduced. Even if the computational cost and energy consumption cost of the push-down processing path increase, as long as the total cost difference is greater than zero, the system determines that the push-down scheme should be adopted, thereby reducing network resource consumption and improving query processing efficiency as a whole.
[0045] V. Alternative Implementation Examples:
[0046] In the dynamic index reconstruction mechanism, drift metric is used in addition to... Besides divergence, Wasserstein distance or population stability index can also be used as alternatives. The divergence calculation of the feature distribution drift remains unchanged, and the technical solutions of generating reconstruction plans, creating virtual snapshots for background asynchronous reconstruction, and performing atomic pointer switching in other control nodes remain the same. This can also achieve the technical effect of avoiding offline full reconstruction and avoiding read-write lock conflicts.
[0047] In the multi-dimensional cost-aware pushdown scheduling mechanism, in addition to generating bytecode or field-programmable gate array instructions through just-in-time compilation and issuing them to the storage-side hardware acceleration unit for execution, a lightweight computing library can also be pre-deployed in the storage node. The control node can directly call the locally deployed lightweight computing library on the storage node to perform pushdown calculations based on the cost model calculation results, without having to perform real-time compilation. This alternative implementation can also achieve the technical effect of operator pushdown and reducing network transmission volume on storage nodes with insufficient hardware acceleration unit resources or without programmability, and can serve as an effective supplement to the present invention in terms of heterogeneous hardware environment adaptation.
[0048] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.
Claims
1. A distributed data processing system based on big data mining, characterized in that, It includes a control node, a compute node cluster, a storage node cluster, and a hardware acceleration unit deployed on the storage node cluster; Each storage node in the storage node cluster is equipped with an incremental data statistics probe, an index structure, and a multi-version concurrency control module. The incremental data statistics probe is used to periodically collect the characteristic distribution of the data shards managed by the storage node according to a preset time window. The control node includes a drift evaluation module and a multidimensional cost optimizer; The drift assessment module is used to calculate the KL divergence relative to the historical benchmark distribution based on the collected feature distribution and the historical benchmark distribution as the drift degree, and to calculate the dynamic threshold based on the mean, standard deviation and pre-set sensitivity coefficient of the drift degree calculated from multiple historical monitoring periods. The drift degree is compared with the dynamic threshold, and when the drift degree is greater than the dynamic threshold, an index reconstruction metadata plan is generated and sent to the corresponding target storage node. The target storage node is used to create a virtual snapshot of the current index structure based on the multi-version concurrency control module after receiving the index reconstruction metadata plan, perform background asynchronous index reconstruction on the virtual snapshot, and after the background asynchronous index reconstruction is completed, switch the read request routing pointer from pointing to the original index structure to pointing to the newly constructed index structure through an atomic pointer switching operation; The multidimensional cost optimizer is used to construct a global cost model based on network transmission cost, computing cost and energy consumption cost. It calculates the total cost of high-cost operators in the query execution path under the path of being pulled up to the computing node for centralized processing and the total cost under the path of being pushed down to the storage node for local processing, and calculates the difference between the two total costs as the push-down benefit. When the pushdown benefit is greater than zero, the control node is used to compile the high-cost operator into instructions that can be executed on the hardware acceleration unit, and send the instructions to the hardware acceleration unit of the target storage node. The hardware acceleration unit is used to perform at least one of the filtering operation, regular expression matching operation and feature extraction operation on the local data, and send the processed valid feature results back to the computing node that initiated the request through the network.
2. The distributed data processing system based on big data mining according to claim 1, characterized in that, The total number of discretized bins for the feature values is preset based on the historical statistical characteristics of data sharding, and the sensitivity coefficient is preset by the system administrator based on business tolerance and can be dynamically adjusted.
3. The distributed data processing system based on big data mining according to claim 1, characterized in that, The virtual snapshot and the original index structure are isolated from each other at the storage level. The background asynchronous index reconstruction process does not occupy lock resources for online read and write requests. The atomic pointer switching operation is a single, indivisible atomic operation, and read requests arriving before and after the switch can receive consistent service.
4. The distributed data processing system based on big data mining according to claim 1, characterized in that, The index structure is one of two types: log structure merged tree variant index and inverted index. When the index structure is log structure merged tree variant index, the background asynchronous index reconstruction reorders and merges the hierarchical data blocks according to the new merging strategy. When the index structure is inverted index, the background asynchronous index reconstruction recalculates the term weights based on the current feature distribution and incrementally updates the inverted list.
5. The distributed data processing system based on big data mining according to claim 1, characterized in that, The control node also includes a real-time telemetry module, which is deployed on the storage node cluster and network link to collect network bandwidth utilization, processor utilization and hardware acceleration unit utilization. In the global cost model, the sum of the weight coefficients corresponding to network transmission cost, computation cost and energy consumption cost is equal to 1, and the multidimensional cost optimizer dynamically adjusts the values of the three weight coefficients based on the data collected by the real-time telemetry module.
6. The distributed data processing system based on big data mining according to claim 1, characterized in that, The multidimensional cost optimizer uses mixed integer linear programming to solve the execution path combination of each operator in the query execution path under given resource constraints, and obtains the total cost of the high-cost operator under the path of being pulled up to the computing node for centralized processing and the total cost under the path of being pushed down to the storage node for local processing.
7. The distributed data processing system based on big data mining according to claim 1, characterized in that, The control node compiles the high-cost operator into two types: bytecode instructions and field-programmable gate array instructions, and sends them to the hardware acceleration unit of the target storage node for execution. The hardware acceleration unit performs calculations on the local data before the data is read from the storage medium and transmitted through the network interface.
8. The distributed data processing system based on big data mining according to claim 1, characterized in that, The storage nodes in the storage node cluster are also deployed with lightweight computing libraries. When the hardware acceleration unit does not have programmability, the control node calls the lightweight computing library deployed on the target storage node to perform pushdown computation in order to achieve local processing of the high-cost operator.
9. A drift-aware distributed index dynamic reconstruction method, applied to a distributed data processing system including a cluster of control nodes and storage nodes, characterized in that, include: In the storage node cluster, the characteristic distribution of the data shards managed by each storage node is periodically collected according to a preset time window. The drift degree is calculated based on the collected feature distribution and the historical baseline distribution, and the dynamic threshold is calculated based on the drift degree of multiple historical monitoring periods. Determine whether the drift degree is greater than the dynamic threshold. If the determination result is no, maintain the current index structure unchanged. If the determination result is yes, generate an index reconstruction metadata plan and send it to the corresponding target storage node. After receiving the index reconstruction metadata plan, the target storage node creates a virtual snapshot for the current index structure based on a multi-version concurrency control mechanism. The virtual snapshot is isolated from the current index structure at the storage level. Perform background asynchronous index rebuilding on the virtual snapshot; After the background asynchronous index reconstruction is completed, the read request routing pointer is switched from pointing to the original index structure to pointing to the newly constructed index structure through an atomic pointer switching operation; The reconstructed feature distribution is then used as the new historical baseline distribution for updating.
10. A multi-dimensional perceptive long-tail query adaptive pushdown scheduling method, applied to a distributed data processing system including control nodes, compute node clusters, and storage node clusters, characterized in that... include: Receive mining query requests initiated by computing nodes, parse the mining query requests, and generate an initial logic execution plan that includes scanning operators, filtering operators, regular expression matching operators, and feature extraction operators; Collect current network bandwidth utilization, processor utilization, hardware acceleration unit utilization, and energy consumption metrics for each storage node and network link; A global cost model is constructed based on the collected metrics, and the total cost of high-cost operators in the initial logical execution plan is calculated under the path of being pulled up to the computing node for centralized processing and under the path of being pushed down to the storage node for local processing. Calculate the difference between the two total costs as the push-down benefit, and determine whether the push-down benefit is greater than zero; If the result is negative, maintain the initial logic execution plan and instruct the storage node to pull up and transmit the pruned data as a whole to the computing node to perform the remaining filtering and mining calculations. If the determination result is yes, the high-cost operator is compiled into instructions that can be executed on the storage-side hardware acceleration unit. The instructions are sent to the hardware acceleration unit of the target storage node to perform local filtering calculations, and the processed effective feature results are sent back to the computing node that initiated the request via the network to perform subsequent mining and aggregation calculations.
