A method and system for summarizing massive data based on task chain and divide-and-conquer method

By dividing massive data into independent subsets in parallel processing and merging results, the storage and computing bottlenecks of traditional methods when processing large-scale data are solved, and efficient and accurate data aggregation is achieved, with good scalability and fault tolerance.

CN118672790BActive Publication Date: 2024-11-08HANGZHOU ANAQI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411154908.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-08
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

Traditional data processing and aggregation methods encounter storage capacity and computing power limitations when processing large-scale data at TB or even PB level, which cannot meet real-time or quasi-real-time data processing needs, and lack the ability to scale horizontally, and cannot flexibly cope with data volume. Growth and change.

Method used

Using a massive data summary method based on task chain and division and governance methods, by dividing massive original data sets into multiple independent data subsets, each subtask is processed in parallel, using a distributed computing environment for parallel processing, and integrating through a merging algorithm Subset results, dynamically adjust resource allocation and segmentation strategies for efficient summary.

Benefits of technology

It significantly improves the data processing speed, makes full use of distributed computing resources, ensures the accuracy and consistency of data processing, has good scalability and fault tolerance, and reduces hardware investment costs and energy consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118672790B_ABST
    Figure CN118672790B_ABST
Patent Text Reader

Abstract

The present invention proposes a method and system for summarizing massive data based on task chain and divide-and-conquer method, belonging to the field of data processing technology; the method comprises: dividing the massive original data set to be processed into multiple independent data subsets according to a preset division rule, each data subset represents a subtask; connecting the subtasks in sequence based on the relevance of the data and the processing order to form a task chain; processing each subtask in the task chain in parallel to obtain the summary result of the data subset; integrating the summary results of the data subsets through a merging algorithm to obtain the global summary result of the entire data set. By reasonably dividing the massive original data set into multiple independent data subsets and forming a task chain, efficient parallel processing can be achieved in a distributed computing environment, greatly shortening the time period of data summarization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention proposes a method and system for aggregating massive data based on task chain and divide-and-conquer method, belonging to the technical field of data processing. Background Art

[0002] With the rapid development of information technology, the acceleration of the global digitalization process has led to an unprecedented explosive growth of data. Various industries and fields are generating massive amounts of data information every day. In the era of big data, data has become a core element driving corporate decision-making, scientific research, social governance and many other fields. However, in the face of this exponentially growing data torrent, how to quickly, accurately and efficiently process and aggregate these massive amounts of data has become a major challenge facing the industry.

[0003] Traditional data processing and aggregation methods are often based on centralized architectures and limited computing resources. They can still cope with small-scale data, but often encounter many bottlenecks when processing large-scale data at the TB or even PB level. First, the limitations of storage capacity and computing power of traditional single servers or small clusters lead to severe limitations in processing speed, which cannot meet real-time or quasi-real-time data processing requirements. Secondly, the transmission and processing of massive data will generate high I / O overhead, which can easily cause system delays or even crashes. In addition, traditional aggregation methods usually do not have good horizontal scalability when processing large-scale data, and cannot flexibly respond to the growth and changes in data volume.

[0004] On the other hand, with the rise of new computing models such as distributed computing, cloud computing, and edge computing, although more powerful computing resources and processing potential are provided, how to design a summary method that can fully utilize the advantages of these new computing models while taking into account data processing efficiency and accuracy, and at the same time effectively solve a series of complex problems such as data partitioning, parallel processing, and result integration, is undoubtedly a technical challenge that needs to be overcome urgently.

[0005] Therefore, developing a new method and technical solution that can quickly and accurately summarize massive amounts of data will not only help promote the advancement of big data processing technology, but also provide strong support for data-driven decision-making in all walks of life. It has extremely important theoretical significance and broad application prospects. Summary of the invention

[0006] The present invention provides a method and system for aggregating massive data based on task chain and divide-and-conquer method, which is used to solve the problems mentioned in the above background technology:

[0007] The present invention proposes a method for aggregating massive data based on task chain and divide-and-conquer method, the method comprising:

[0008] S1. According to the preset division rules, the massive original data set to be processed is divided into multiple independent data subsets, each data subset represents a subtask;

[0009] S2. Based on the relevance of the data and the processing order, the subtasks are connected in order to form a task chain;

[0010] S3, performing parallel processing on each subtask in the task chain to obtain a summary result of the data subset;

[0011] S4. Integrate the summary results of the data subsets through the merging algorithm to obtain the global summary results of the entire data set.

[0012] Furthermore, the S1 includes:

[0013] S11, dividing the massive original data set into multiple partitions, preprocessing the data in each partition, and after the preprocessing is completed, aggregating the data in each partition to form a complete data set after preprocessing;

[0014] S12, according to data characteristics and computing resources, a divide-and-conquer rule is preset, and according to the preset divide-and-conquer rule, a data segmentation operation is performed to divide the original data set into multiple independent data subsets;

[0015] S13. Define each generated data subset as an independent subtask, and each subtask is executed as a separate work unit in a distributed computing environment.

[0016] Furthermore, the S12 includes:

[0017] Use statistical methods and machine learning algorithms to extract features from massive raw data sets and identify key dimensions and outliers in the data;

[0018] Analyze the dependencies between elements in the data set, and evaluate the resource requirements and potential performance bottlenecks of different segmentation strategies based on currently available computing resources and network conditions;

[0019] Design a multi-dimensional segmentation strategy based on data characteristics, and dynamically adjust the segmentation granularity according to real-time processing performance and data flow changes;

[0020] Use heuristic algorithms or reinforcement learning algorithms to predict and optimize the load distribution of the segmented data subsets in a distributed environment;

[0021] Before data segmentation, parallel computing is used to preprocess the data; based on the preset divide-and-conquer rules and the data characteristics of real-time analysis, the intelligent segmentation algorithm is used to segment the preprocessed data;

[0022] Create independent packaging files for each generated data subset and generate indexing mechanisms;

[0023] Evaluate the processing efficiency, resource utilization, and independence of subtasks of the segmented data subsets in a distributed computing environment through simulation or small-scale testing;

[0024] According to the evaluation results, the divide-and-conquer rules and segmentation strategies are iteratively optimized, and the optimized divide-and-conquer rules and segmentation strategies are incorporated into the knowledge base.

[0025] Furthermore, the dependencies between the elements in the data set are analyzed, and the resource requirements and potential performance bottlenecks of different segmentation strategies are evaluated in combination with the currently available computing resources and network conditions, including:

[0026] Uniquely identify and label each element in a massive data set, and use causal reasoning algorithms to analyze the direct and indirect dependencies between elements and build a data dependency graph;

[0027] Quantitatively evaluate the identified dependencies, build a resource usage model based on the currently available computing resources and network conditions, and simulate the resource requirements of different data processing tasks;

[0028] Based on the data dependency graph, multiple data segmentation strategies are preset. The resource usage model is used to predict the execution performance of each segmentation strategy under different resource configurations and identify potential performance bottlenecks.

[0029] Monitor the dynamic changes of computing resources and network conditions in real time, dynamically evaluate the adaptability and effectiveness of the current segmentation strategy based on environmental change data, and automatically trigger the adjustment of the segmentation strategy according to the evaluation results through the feedback and adjustment mechanism.

[0030] Furthermore, the real-time monitoring of dynamic changes in computing resources and network conditions, dynamically evaluating the adaptability and effectiveness of the current segmentation strategy based on environmental change data, and automatically triggering the adjustment of the segmentation strategy according to the evaluation results through a feedback and adjustment mechanism include:

[0031] Through the real-time monitoring system, the computing resources and network status data are collected in real time; and the collected raw data is pre-processed;

[0032] Use time series analysis to analyze historical data on computing resources and network conditions, predict changing trends, and use anomaly detection mechanisms to monitor abnormal fluctuations in resource usage or network conditions;

[0033] Compare and analyze the environmental change data with the preset performance indicators to evaluate the adaptability of the current segmentation strategy in different environments;

[0034] According to the results of adaptability analysis, set the strategy that triggers the segmentation strategy adjustment, and build an adjustment strategy library containing multiple alternative segmentation strategies;

[0035] Through the intelligent decision-making engine, the segmentation strategy is selected from the adjustment strategy library for adjustment according to the real-time environmental change data and preset trigger conditions;

[0036] The effectiveness of the adjusted segmentation strategy is evaluated and the evaluation results are fed back to the real-time monitoring and adaptive evaluation links to form a closed-loop optimization mechanism.

[0037] Furthermore, the multi-dimensional segmentation strategy is designed based on data characteristics, and the segmentation granularity is dynamically adjusted according to real-time processing performance and data flow changes, including:

[0038] Evaluate the density distribution of data in different regions or dimensions, identify sparse and dense areas, establish a real-time data flow monitoring system, and capture the dynamic changes of data flow;

[0039] Based on the results of data characteristic analysis, the overall framework of multi-dimensional segmentation strategy is constructed to evaluate the contribution of different dimensions to the segmentation effect and optimize the selection of segmentation dimensions;

[0040] Through the adaptive segmentation granularity adjustment mechanism, the segmentation granularity is dynamically adjusted according to the real-time processing performance and data flow changes.

[0041] Furthermore, the S2 includes:

[0042] S21, analyzing the logical correlation between each data subset, determining the front-to-back dependency relationship between subtasks according to the analysis results, and constructing a task dependency graph;

[0043] S22. Design the order of task execution based on the identified dependencies and form a linear task chain or a complex directed acyclic graph structure;

[0044] S23, formulate a task scheduling strategy for task chain execution, encapsulate subtasks into executable units, and configure input data and output paths;

[0045] S24. Implement a task execution monitoring system to track the execution status of the task chain and establish a task coordination mechanism;

[0046] S25. When all subtasks are completed in sequence, their respective results are integrated to form a complete task chain.

[0047] Furthermore, the S3 includes:

[0048] S31, initialize the distributed computing environment, deploy and start the task execution framework, and load each subtask in the task chain into the computing resource pool;

[0049] S32, start each subtask to process its corresponding data subset in parallel, each subtask is executed independently on its computing node, and generates a corresponding subset summary result;

[0050] S33, setting a resource monitoring module to continuously monitor the computing resources occupied by each subtask during execution, and reporting resource usage data to the resource scheduling center;

[0051] S34, based on the collected resource usage, a load balancing algorithm is used to evaluate whether the current system resource allocation is balanced; the resource scheduler dynamically adjusts the computing resource allocation of each subtask based on the evaluation result of the load balancing algorithm;

[0052] S35, after the parallel processing of the subtasks is completed, the summary results of each subtask are integrated together through the synchronization mechanism to form a preliminary summary of the entire original data set;

[0053] S36. Based on the resource scheduling effect during the actual execution process, continuously provide feedback to optimize the resource scheduling strategy.

[0054] Furthermore, the S34 includes:

[0055] Collect multi-dimensional resource usage data, and use machine learning to identify performance bottlenecks in the current system based on the collected multi-dimensional resource usage data;

[0056] Construct a load pressure assessment model to evaluate the load pressure of each computing node based on multiple factors;

[0057] Through the load balancing algorithm based on multi-objective optimization and combined with the resource usage evaluation results, the parameters and strategies in the load balancing algorithm are dynamically adjusted;

[0058] The resource scheduler generates resource scheduling decisions based on the evaluation results of the load balancing algorithm and the dynamic adjustment strategy, and sends the generated resource scheduling decisions to each computing node to perform corresponding task migration and resource reallocation operations;

[0059] By monitoring the system status and resource usage after resource scheduling, the scheduling effect is evaluated, and based on the evaluation results, continuous feedback is provided to optimize the resource scheduling strategy.

[0060] Furthermore, the S4 includes:

[0061] S41. Select the merging algorithm based on the data type and specific requirements of the aggregation task, and clarify the specific implementation details of the merging algorithm;

[0062] S42, creating a data structure for storing global summary results, and after completing the processing of all subtasks, collecting the summary results of each subset from each computing node, and transmitting them to the master control node or the designated merge node;

[0063] S43, executing a merging algorithm, and gradually integrating the summary results of each subset according to a preset merging rule to form a complete global summary result;

[0064] S44, performing consistency check on the merged global summary results. If there is a loss of accuracy or a large deviation, fine-tune and optimize the merging algorithm and perform a secondary merge.

[0065] S45. Collect feedback information based on the performance and result quality during the merging process, and continuously optimize the merging algorithm and data processing flow based on the feedback information.

[0066] The present invention proposes a system for implementing a method for summarizing massive data based on task chains and divide-and-conquer method, the system comprising:

[0067] Data partitioning module: divides the massive original data set to be processed into multiple independent data subsets according to the preset partitioning rules, each of which represents a subtask;

[0068] Task chain forming module: based on the relevance of data and the processing order, the subtasks are connected in order to form a task chain;

[0069] Data aggregation module: each subtask in the task chain is processed in parallel to obtain the aggregation result of the data subset;

[0070] Result integration module: integrates the summary results of data subsets through a merging algorithm to obtain the global summary results of the entire data set.

[0071] Beneficial effects of the present invention: By reasonably dividing a massive original data set into multiple independent data subsets and forming a task chain, efficient parallel processing can be achieved in a distributed computing environment, greatly shortening the time cycle of data aggregation. Data preprocessing and partitioning help reduce invalid calculations and improve data quality. At the same time, divide-and-conquer rules are set according to data characteristics and computing resources to ensure the rationality of data segmentation and effective use of system resources. By analyzing the logical correlation between subtasks, forming a task dependency graph and a task chain, the order and dependency of task execution can be accurately controlled to ensure the correctness and integrity of the data processing flow. In the parallel processing process, by real-time monitoring of resource usage and applying a load balancing algorithm, the computing resource allocation of each subtask is dynamically adjusted, which not only improves the operating efficiency of the overall system, but also ensures the stability of task execution. The appropriate merging algorithm is selected in combination with the data type and the requirements of the aggregation task to ensure that the local results of all subsets can be accurately integrated in the aggregation stage to form a global aggregation result, and the result quality and accuracy are further improved through consistency verification and fine-tuning optimization. The entire process includes a large number of feedback mechanisms. Whether it is monitoring the execution of subtasks or evaluating the effectiveness of merging algorithms, key performance indicators and problem feedback can be collected to continuously optimize the entire data processing process and algorithm strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of the method of the present invention;

[0073] Figure 2 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0074] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0077] One embodiment of the present invention, as Figure 1 As shown, a method for summarizing massive data based on task chain and divide-and-conquer method, the method comprising:

[0078] S1. According to the preset division rules, the massive original data set to be processed is divided into multiple independent data subsets, each data subset represents a subtask;

[0079] S2. Based on the relevance of the data and the processing order, the subtasks are connected in order to form a task chain;

[0080] S3, performing parallel processing on each subtask in the task chain to obtain a summary result of the data subset;

[0081] S4. Integrate the summary results of the data subsets through the merging algorithm to obtain the global summary results of the entire data set.

[0082] The working principle of the above technical solution is as follows: First, according to the preset division rules, the massive original data set to be processed is intelligently divided into multiple independent and related data subsets. Each data subset represents an independent processing unit, namely a subtask. Such a divide-and-conquer strategy greatly reduces the pressure on a single computing node, and also conforms to the concept of distributed computing, making big data processing more efficient and feasible; based on the correlation and processing order between data subsets, each subtask is connected in series according to a logical relationship to form one or more task chains. The construction of the task chain fully considers the continuity and dependency of data processing, and ensures the correctness and integrity of data processing. The output of each subtask will be used as the input of the next subtask, forming an orderly, pipelined processing flow; each subtask in the task chain is processed in a parallel computing manner. Through effective resource scheduling strategies, such as load balancing algorithms, each subtask is reasonably allocated to different computing resources to achieve concurrent execution, which greatly improves the data processing speed. During the parallel processing, the computing resource load of each subtask is monitored in real time, and resource allocation is adjusted dynamically to ensure the optimal overall operating efficiency of the system; when all subtasks complete their respective processing tasks, the summary results of each data subset are obtained. Next, the summary results of each subset are integrated through a pre-selected merging algorithm (such as distributed Reduce operation, merge sort, etc.). This merging process can ensure that the global summary results of the entire data set are accurately restored from a large number of scattered local summary results, thereby meeting business needs and data analysis purposes.

[0083] The effect of the above technical solution is: by dividing the original data set into multiple independent data subsets and processing each subtask in parallel, the data processing speed is significantly improved, and the problem of huge data volume in the big data environment is effectively dealt with. This method gives full play to the advantages of distributed computing and parallel processing, so that the summary work that may take a long time to complete can be completed quickly; based on the task chain and divide-and-conquer strategy, the task allocation and resource scheduling can be dynamically adjusted according to the size of the data subset and the computing resource status to achieve load balancing. This can avoid overloading of a single computing node and make full use of the overall computing power of the cluster, thereby improving the system throughput and response speed; by building a task chain based on the relevance and processing order of the data, it is ensured that the correct data dependency can be followed even in the process of parallel processing, ensuring the accuracy and consistency of the summary results; the divide-and-conquer method makes the processing process have good scalability. When the amount of data increases or the computing resources increase, only more subtasks and computing nodes need to be added without making major changes to the entire processing flow. At the same time, since each subtask is relatively independent, fault recovery and system maintenance are also more convenient; compared with processing the entire original data set at one time, the divide-and-conquer and parallel processing methods can save computing resources, shorten processing time, and indirectly reduce hardware investment costs and energy consumption, which has a positive impact on long-term operations and green energy conservation; through the pre-set merging algorithm, the summary results of each data subset can be accurately merged into the global summary result of the entire data set, ensuring that the summary result fully reflects the information of the original data set.

[0084] In one embodiment of the present invention, the S1 includes:

[0085] S11, dividing the massive original data set into multiple partitions, preprocessing the data in each partition, and after the preprocessing is completed, aggregating the data in each partition to form a complete data set after preprocessing;

[0086] S12. Preset a divide-and-conquer rule based on data characteristics and computing resources, such as dividing according to factors such as time series, geographical area, user identification, or using a uniform division method such as a hash function; perform a data division operation according to the preset divide-and-conquer rule to divide the original data set into multiple independent data subsets;

[0087] S13. Define each generated data subset as an independent subtask, and each subtask is executed as a separate work unit in a distributed computing environment.

[0088] The working principle of the above technical solution is as follows: First, the original massive data set is divided into multiple partitions according to certain logical or physical attributes. These partitions are usually determined based on the intrinsic structural characteristics of the data, such as time series, geographic location, user identification or other related attributes, so as to facilitate management and processing. In each partition, the data is preprocessed as necessary, which may include operations such as cleaning, conversion, standardization, and deduplication to ensure the accuracy and efficiency of subsequent analysis or calculation. After the preprocessing is completed, the data in the partition is locally summarized to form a complete preprocessed data set in the partition. According to the data characteristics and available computing resources, a preset divide-and-conquer rule is designed. For example, if the data has obvious time linear characteristics, it may be divided by time window; if it is spatial data, it may be divided by geographic region; if it is user behavior data, it may be divided by user identification. In addition, a uniform partitioning strategy, such as a hash function, can also be used to randomly distribute the data into different subsets to ensure that the amount of data in each subset is relatively balanced. According to the preset divide-and-conquer rule, the data partitioning operation is actually performed to cut the original data set into multiple non-overlapping and independent data subsets, each of which contains some of the features of a part of the original data. Each data subset obtained by division is regarded as an independent subtask. In a distributed computing environment (such as Hadoop, Spark and other frameworks), each subtask is encapsulated into a work unit, which is often called a task or job. The distributed computing environment will automatically assign these subtasks to different computing nodes for parallel processing. Each computing node is only responsible for the data aggregation of its corresponding subtask, which can greatly reduce the overall processing time. After each subtask is executed, the local summary results generated by each of them will be further integrated, and through a preset merging algorithm (such as Reduce operation), the global summary result of the entire original data set will be finally obtained.

[0089] The effect of the above technical solution is: by dividing the massive original data set into multiple small data subsets and processing them separately in a distributed computing environment, the advantages of parallel computing of multiple computers or multiple computing nodes in the cluster can be fully utilized, and the pressure of single-point processing can be significantly reduced, thereby greatly improving the data processing speed and overall efficiency; according to the data characteristics and computing resources, the preset divide-and-conquer rules can be used to allocate computing resources more reasonably. For example, data based on time series can be dispersed into different time periods for processing in the form of time windows, avoiding excessive concentration of computing resources at a certain moment. Uniform partitioning methods (such as hash functions) help to ensure that the amount of data for each subtask is roughly the same, so that the load of each computing node is balanced, effectively preventing the "hot spot" problem, and improving the stability and processing capacity of the system. Each data subset runs as an independent subtask, which enhances the modularity of the system, so that new subtasks can be easily added for processing when new data is added, and it is also convenient for system maintenance and upgrades. Data partitioning can be customized according to business needs and data characteristics. For example, partitioning by geographical area helps to achieve regional-level data mining and analysis; partitioning by user ID is conducive to scenarios such as personalized recommendation or user behavior analysis, making data analysis more accurate and targeted. Breaking down a big problem into smaller ones reduces the complexity of a single task and makes each subtask easier to understand and handle. At the same time, the relative independence of each subtask also reduces the potential propagation of errors and the scope of impact. In a distributed environment, even if a subtask fails, the failure recovery process of the entire system is faster and more convenient due to the independence of other subtasks. Only the failed subtask needs to be reprocessed without affecting the results of the successfully processed part.

[0090] In one embodiment of the present invention, the S12 includes:

[0091] Use statistical methods and machine learning algorithms to extract features from massive raw data sets and identify key dimensions and outliers in the data;

[0092] Analyze the dependencies between elements in the data set, and evaluate the resource requirements and potential performance bottlenecks of different segmentation strategies based on currently available computing resources and network conditions;

[0093] Design multi-dimensional segmentation strategies based on data characteristics, such as hybrid segmentation combining time windows and geographic locations, or cluster segmentation based on data similarity;

[0094] Introduce monitoring and feedback mechanisms to dynamically adjust the segmentation granularity based on real-time processing performance and data flow changes; automatically increase the segmentation granularity when encountering a computing bottleneck;

[0095] Use heuristic algorithms or reinforcement learning techniques to predict and optimize the load distribution of the segmented data subsets in a distributed environment;

[0096] Before data segmentation, parallel computing technology is used to pre-process the data; based on the preset divide-and-conquer rules and the data characteristics of real-time analysis, the intelligent segmentation algorithm is used to accurately segment the pre-processed data;

[0097] Create independent encapsulation files for each generated data subset, and generate an efficient indexing mechanism to facilitate subsequent rapid positioning and processing in a distributed environment.

[0098] Through simulation or small-scale testing, the processing efficiency, resource utilization, and independence of subtasks of the divided data subsets in a distributed computing environment are evaluated.

[0099] According to the evaluation results, the divide-and-conquer rules and segmentation strategies are iteratively optimized until the preset performance indicators are reached or the optimization converges; the optimized divide-and-conquer rules and segmentation strategies are incorporated into the knowledge base.

[0100] The working principle of the above technical solution is: use statistical methods and machine learning algorithms to conduct in-depth analysis of massive raw data sets, extract key data features (such as time series patterns, spatial distribution laws, category distribution, etc.) and identify outliers; conduct detailed analysis of the dependencies between elements in the data set, especially the temporal dependencies of time series data and the node connection relationships in graph data, so as to understand the inherent structure and complexity of the data. Before data segmentation, parallel computing technology is used to preprocess the data set, including data cleaning (removing noise and irrelevant data), standardization (unifying data format and dimensions), compression (reducing storage and transmission costs), etc., to reduce the complexity and time consumption of subsequent processing. Design appropriate divide-and-conquer rules based on data characteristics and currently available computing resources (CPU, GPU, memory, storage, etc.) and network conditions. These rules are designed to optimize data segmentation strategies to make full use of computing resources and reduce potential performance bottlenecks. Design multi-dimensional segmentation strategies, such as hybrid segmentation combining time windows and geographic locations, or cluster segmentation based on data similarity, to improve the independence and processing efficiency between subtasks. Evaluate the resource requirements and potential performance impact of different segmentation strategies to provide decision-making basis for subsequent segmentation operations. Introduce monitoring and feedback mechanisms to track data processing performance and data flow changes in real time. When encountering computing bottlenecks or performance degradation, automatically adjust the segmentation granularity to balance the load of each subtask and reduce the burden of a single subtask. Apply intelligent segmentation algorithms (such as graph-based algorithms, K-means clustering algorithms, etc.) to accurately segment the preprocessed data according to the preset divide-and-conquer rules and real-time analysis of data characteristics. These algorithms can automatically identify and divide similarities and differences in data to generate independent and efficient data subsets. Create independent package files for each generated data subset to ensure that they can be processed and transmitted independently in a distributed environment. Generate an efficient indexing mechanism to facilitate rapid location and processing of data subsets in subsequent processing. These indexes can be built based on the key features of the data or segmentation strategies to improve the efficiency and accuracy of data access. Through simulation or small-scale testing, evaluate the processing efficiency, resource utilization, and independence between subtasks of the segmented data subsets in a distributed computing environment. These evaluation results help understand the effectiveness and potential problems of the current segmentation strategy. Based on the evaluation results, iteratively optimize the divide-and-conquer rules and segmentation strategies. By adjusting the segmentation granularity, improving the segmentation algorithm, or optimizing resource allocation, the preset performance indicators can be gradually approached or optimization convergence can be achieved. The optimized divide-and-conquer rules and segmentation strategies are incorporated into the knowledge base to provide experience and reference for the subsequent processing of similar tasks. This helps to achieve knowledge accumulation and inheritance, and improve the intelligence and automation level of data processing.

[0101] The effects of the above technical solutions are as follows: through accurate data feature extraction and dependency analysis, a segmentation strategy that is more in line with data characteristics can be designed, thereby avoiding unnecessary data redundancy and repeated processing, and significantly improving the overall efficiency of data processing; the introduction of parallel computing technology for data preprocessing can process a large amount of data at the same time, significantly shortening the data preparation time and laying the foundation for subsequent segmentation operations; the application of intelligent segmentation algorithms, such as graph-based partitioning algorithms, K-means clustering algorithms, etc., can automatically identify and divide similarities and differences in data, achieve efficient data segmentation, and reduce processing time; combined with computing resources (CPU, GPU, memory, storage, etc.) and network conditions, the resource requirements and potential performance bottlenecks of different segmentation strategies are evaluated to ensure that the segmentation strategy can make full use of existing resources and avoid resource waste; heuristic algorithms or reinforcement learning technologies are used to optimize load distribution to ensure load balancing of nodes in a distributed environment, improve overall resource utilization, and reduce performance bottlenecks caused by uneven resources; multi-dimensional segmentation strategies Designs, such as hybrid segmentation combining time windows and geographic locations, or cluster segmentation based on data similarity, can effectively improve the independence between subtasks, reduce data dependence and communication costs between subtasks, and facilitate distributed parallel processing; this independence also makes it easier for subtasks to be executed independently on different computing nodes, improving the fault tolerance and scalability of the system; the introduction of monitoring and feedback mechanisms enables the system to dynamically adjust the segmentation granularity according to real-time processing performance and data flow changes, enhancing the system's adaptability and flexibility; by iteratively optimizing divide-and-conquer rules and segmentation strategies, the system can continuously learn and evolve to cope with data sets of different sizes and characteristics, improving the system's versatility and scalability; the optimized divide-and-conquer rules and segmentation strategies are incorporated into the knowledge base to provide experience and reference for the subsequent processing of similar tasks, promoting the accumulation and inheritance of knowledge; this knowledge base can not only provide optimization guidance for the current system, but also provide valuable reference and reference for future data processing tasks, promoting the continuous advancement and development of data processing technology.

[0102] In one embodiment of the present invention, the dependency relationship between the elements in the data set is analyzed, and the resource requirements and potential performance bottlenecks of different segmentation strategies are evaluated in combination with currently available computing resources and network conditions, including:

[0103] Uniquely identify and label each element (such as features, variables, records, etc.) in massive data sets, and use causal reasoning algorithms to analyze the direct and indirect dependencies between elements and build a data dependency graph;

[0104] Quantitatively evaluate the identified dependencies, build a resource usage model based on the currently available computing resources (CPU, memory, storage, network bandwidth, etc.) and network conditions (latency, packet loss rate, etc.), and simulate the resource requirements of different data processing tasks;

[0105] Based on the data dependency graph, multiple data segmentation strategies are preset, including horizontal segmentation, vertical segmentation, mixed segmentation, etc., and assumptions are made about how each strategy is applied to the data set. The resource usage model is used to predict the execution performance of each segmentation strategy under different resource configurations and identify potential performance bottlenecks, such as memory overflow, network congestion, and uneven computing load.

[0106] Monitor the dynamic changes of computing resources and network conditions in real time, dynamically evaluate the adaptability and effectiveness of the current segmentation strategy based on environmental change data, and automatically trigger the adjustment of the segmentation strategy according to the evaluation results through the feedback and adjustment mechanism.

[0107] The working principle of the above technical solution is: uniquely identify each element in the data set (such as features, variables, records, etc.) and assign tags for subsequent tracking and analysis; use causal reasoning algorithms or correlation analysis techniques to analyze direct and indirect dependencies between elements. These relationships can be based on various factors such as statistical correlation, causal chains, business logic, etc.; visualize the analyzed dependencies in the form of a graph, and focus on marking strong dependencies, which will play a key role in subsequent data segmentation and processing; measure the strength and importance of dependencies by calculating quantitative indicators such as correlation coefficients and information gain, and provide a scientific basis for data segmentation strategies; build a resource usage model based on currently available computing resources (CPU, memory, storage, network bandwidth, etc.) and network conditions (delay, packet loss rate, etc.). The model can simulate the resource requirements of different data processing tasks and predict resource consumption. Based on the data dependency graph, it presets multiple data segmentation strategies, including horizontal segmentation (by record), vertical segmentation (by field), mixed segmentation, etc., and clarifies how each strategy is applied to the data set. It uses the resource usage model to predict the execution performance of different segmentation strategies under different resource configurations, including key indicators such as processing speed and resource utilization. At the same time, it identifies potential performance bottlenecks, such as memory overflow, network congestion, and uneven computing load. It continuously monitors the dynamic changes of computing resources and network conditions, including real-time data of indicators such as resource utilization, load fluctuation, and network latency. Based on real-time monitoring data, it dynamically evaluates the adaptability and effectiveness of the current segmentation strategy. It identifies performance degradation or resource waste caused by environmental changes, such as insufficient or overloaded resource utilization. Based on the evaluation results, it automatically triggers the adjustment of the segmentation strategy. By optimizing segmentation strategies, adjusting resource allocation or introducing new processing mechanisms to cope with resource fluctuations and data flow changes, ensure the efficient and stable operation of the system; collect user feedback and system operation data, and analyze problems and bottlenecks in the data segmentation and processing process; based on feedback data, continuously optimize data dependency analysis algorithms, resource usage models and segmentation strategy design to improve system performance and resource utilization.

[0108] The effect of the above technical solution is: by analyzing the dependencies between data elements through causal reasoning algorithms and constructing data dependency graphs, the key dependencies and independent parts in the data can be more accurately identified. This helps to design a more reasonable data segmentation strategy, reduce unnecessary data redundancy and repeated processing, and thus improve the overall efficiency of data processing; by building a resource usage model, the requirements of different data processing tasks for computing resources (CPU, memory, storage, network bandwidth, etc.) and network conditions can be simulated. This helps to evaluate the resource requirements of different segmentation strategies in advance, thereby avoiding over-allocation or under-allocation of resources and improving the overall utilization of resources; using resource usage models to predict the execution performance of different segmentation strategies under different resource configurations, potential performance bottlenecks can be identified, such as memory overflow, network congestion, and uneven computing load. By identifying and avoiding these bottlenecks in advance, it can ensure that the system can still maintain stable and efficient operation under high load and complex environments; real-time monitoring of the dynamic changes in computing resources and network conditions, and dynamically evaluating the adaptability and effectiveness of the current segmentation strategy based on environmental change data. This dynamic adjustment mechanism enables the system to quickly respond to resource fluctuations and data flow changes, automatically triggering the adjustment of the segmentation strategy, thereby maintaining efficient and stable operation of the system. This flexibility is particularly important for processing large-scale, dynamically changing data sets; quantitative evaluation of identified dependencies, such as measuring the strength and importance of dependencies through indicators such as correlation coefficients and information gain, provides a scientific basis for the subsequent segmentation strategy design. This makes the decision-making process more scientific and reasonable, helps to formulate more effective data processing strategies, and improves the support capabilities for business decisions; by optimizing data processing processes, improving resource utilization, and reducing performance bottleneck risks, the cost and risk of data processing can be further reduced. This helps enterprises achieve more efficient operations management and enhance competitiveness.

[0109] In one embodiment of the present invention, the quantitative evaluation of the identified dependencies, building a resource usage model based on currently available computing resources and network conditions, and simulating resource requirements of different data processing tasks include:

[0110] Evaluate the strength of dependencies between elements through complex network analysis algorithms (such as PageRank and HITS algorithm variants); assign a weight value to each pair of dependencies; and divide dependencies into different types of dependencies, which have significant differences in resource requirements; further adjust the weight according to the dependency type;

[0111] Identify and quantify key characteristics of computing resources such as CPU model, number of cores, cache size, GPU support, and how key characteristics of computing resources affect the performance of specific data processing tasks;

[0112] Evaluate storage device characteristics, including type (SSD, HDD), capacity, IOPS (input and output operations per second), latency, and analyze their impact on data read and write speeds;

[0113] Based on network factors, including network bandwidth, latency, and packet loss rate, a network status model is established to simulate resource consumption and potential bottlenecks during data transmission;

[0114] Decompose the data processing task into multiple subtasks, and map each subtask to specific computing, storage, and network resources based on the dependency graph and subtask characteristics;

[0115] Using the cloud computing simulation platform, based on the extracted resource characteristics and task decomposition results, the execution process of different data processing tasks under different resource configurations is simulated;

[0116] Through simulation results, predict the key performance indicators of each task configuration and identify potential performance bottlenecks;

[0117] By adjusting the resource allocation ratio, sensitivity analysis is performed based on changes in performance indicators, and resource allocation is dynamically adjusted according to real-time resource usage and task requirements, ensuring efficient use of resources and rapid completion of tasks.

[0118] The working principle of the above technical solution is: using complex network analysis algorithms such as PageRank and HITS algorithm variants to quantitatively evaluate the dependencies between elements in massive data sets. Considering the directness, frequency and length of the dependency chain, a weight value is assigned to each pair of dependencies; the dependencies are further divided into data reading dependency, calculation dependency, storage dependency and communication dependency. Different types of dependencies have significantly different requirements for resources, so the weights need to be further adjusted according to the dependency type; identify and quantify key features of computing resources such as CPU model, number of cores, cache size, GPU support, and analyze the performance impact of these features on specific data processing tasks; evaluate storage device types (SSD, HDD), capacity, IOPS (input and output operations per second), latency and other features, and analyze their impact on data reading and writing speeds; build a network status model based on network factors such as network bandwidth, latency, and packet loss rate to simulate resource consumption and potential bottlenecks during data transmission; decompose data processing tasks into multiple subtasks to manage resource allocation more meticulously; map each subtask to specific computing, storage, and network resources based on the dependency graph and subtask characteristics to ensure effective use of resources; use the cloud computing simulation platform to simulate the execution process of different data processing tasks under different resource configurations based on the extracted resource features and task decomposition results; predict the key performance indicators of each task configuration through simulation results, including execution time, throughput, resource utilization, etc. At the same time, potential performance bottlenecks such as resource contention and network congestion are identified; by adjusting the resource allocation ratio (such as increasing the number of CPU cores, expanding memory capacity, optimizing network bandwidth, etc.), changes in performance indicators are observed and sensitivity analysis is performed. This helps determine the optimal direction and magnitude of resource adjustment; based on real-time resource usage and task requirements, resource allocation is dynamically adjusted. Through the feedback mechanism, resource allocation is continuously optimized to ensure efficient resource utilization and rapid task completion; the real-time monitoring system continuously collects data on computing resources and network conditions, and continuously evaluates resource usage and task execution effects; based on the evaluation results, the resource usage model, task decomposition strategy and resource adjustment mechanism are continuously optimized to adapt to the changing environment and task requirements.

[0119] The effects of the above technical solutions are as follows: the intensity of dependencies between elements is quantified through complex network analysis algorithms, and weights are assigned considering multiple factors (such as directness, frequency, and dependency chain length), making resource demand assessment more accurate; dependencies are subdivided into multiple types such as data reading, computing, storage, and communication, and weights are adjusted for different types, further improving the accuracy of resource demand assessment; key features of computing resources (such as CPU, GPU), storage devices, and network factors are identified and quantified, providing comprehensive and accurate data support for the construction of resource usage models; the constructed resource usage model can simulate the execution process and predict performance indicators based on different data processing tasks and resource configurations, thereby improving the adaptability and flexibility of the model; data processing tasks are decomposed into subtasks, and mapped to specific resources according to dependency graphs and subtask characteristics, realizing refined resource management; through simulation results and sensitivity analysis, the resource allocation ratio can be adjusted in real time, To cope with potential performance bottlenecks such as resource contention and network congestion, ensure efficient use of resources and rapid completion of tasks; the simulation process can predict the performance of key indicators under different task configurations in advance, identify potential performance bottlenecks such as resource contention and network congestion, and provide a time window for problem solving; based on simulation results and performance predictions, targeted optimization strategies can be formulated, such as adjusting resource allocation, optimizing task decomposition, etc., to improve the overall performance and stability of the system; through accurate resource demand assessment and efficient resource allocation, resource waste and idleness are reduced, and operating costs are reduced; the rapid completion and efficient execution of tasks improve the company's production efficiency and business response speed, and enhance market competitiveness; this technical solution can adapt to different data processing scenarios and task requirements, and has a wide range of applicability; with the continuous development of computing resources, storage devices and network technologies, this technical solution can be easily expanded and upgraded to support more complex data processing tasks and higher performance requirements.

[0120] In one embodiment of the present invention, the real-time monitoring of dynamic changes in computing resources and network conditions, dynamically evaluating the adaptability and effectiveness of the current segmentation strategy based on environmental change data, and automatically triggering the adjustment of the segmentation strategy according to the evaluation results through a feedback and adjustment mechanism include:

[0121] Through the real-time monitoring system, the computing resources (such as CPU usage, memory usage, GPU load) and network status (such as bandwidth usage, delay, packet loss rate) data are collected in real time; and the collected raw data is pre-processed;

[0122] Use time series analysis to analyze historical data on computing resources and network conditions, predict changing trends, and use anomaly detection mechanisms to monitor abnormal fluctuations in resource usage or network conditions;

[0123] Compare and analyze the environmental change data with the preset performance indicators to evaluate the adaptability of the current segmentation strategy in different environments;

[0124] According to the results of adaptability analysis, set the strategy that triggers the segmentation strategy adjustment, such as resource utilization exceeding the threshold, network delay significantly increased, etc.; build an adjustment strategy library containing multiple alternative segmentation strategies;

[0125] Through the intelligent decision-making engine, the segmentation strategy is selected from the adjustment strategy library for adjustment according to the real-time environmental change data and preset trigger conditions;

[0126] The effectiveness of the adjusted segmentation strategy is evaluated and the evaluation results are fed back to the real-time monitoring and adaptive evaluation links to form a closed-loop optimization mechanism.

[0127] The working principle of the above technical solution is: deploy one or more real-time monitoring systems to continuously collect real-time data of computing resources (such as CPU usage, memory usage, GPU load) and network conditions (such as bandwidth usage, delay, packet loss rate); perform pre-processing operations such as cleaning, denoising, and normalization on the collected raw data to ensure the accuracy and availability of the data; use time series analysis technology to analyze historical data of computing resources and network conditions, identify seasonal and periodic patterns in the data, and predict future trends; monitor abnormal fluctuations in resource usage or network conditions through statistical methods or machine learning algorithms. When the data deviates from the normal range, an abnormal alarm is triggered; compare and analyze the real-time collected environmental change data with the preset performance indicators. These performance indicators may include but are not limited to execution time, throughput, resource utilization, etc.; based on the results of the comparative analysis, evaluate the adaptability of the current segmentation strategy in different environments. Determine whether the current strategy can meet the performance requirements or whether there is a better segmentation strategy option; set the conditions for triggering the adjustment of the segmentation strategy based on the results of the adaptability analysis. These conditions may include resource utilization exceeding the threshold, network delay significantly increased, etc.; build an adjustment strategy library containing multiple alternative segmentation strategies. Each strategy is pre-evaluated and optimized to adapt to different environmental changes. The strategy library should be updated regularly to include the latest optimization results and strategies to adapt to new environments; based on real-time environmental change data and preset trigger conditions, the intelligent decision engine selects the most appropriate segmentation strategy from the adjustment strategy library for adjustment. This selection process may involve complex decision logic and algorithm optimization; under the guidance of the intelligent decision engine, the adjustment of the segmentation strategy is automatically triggered. Including operations such as reallocating resources, adjusting task priorities, and optimizing data transmission paths; evaluating the effects of the adjusted segmentation strategy and monitoring changes in performance indicators. The evaluation results should be able to reflect the actual effect of the strategy adjustment; the evaluation results are fed back to the real-time monitoring and adaptability evaluation links to form a closed-loop optimization mechanism. Based on the feedback results, the monitoring strategy, prediction model, adaptability evaluation method, and adjustment strategy library are continuously adjusted and optimized to improve the overall performance and stability of the system.

[0128] The effects of the above technical solutions are as follows: through the real-time monitoring system, the dynamic changes of computing resources and network conditions can be instantly perceived to ensure that the system can respond quickly to environmental changes; based on real-time monitoring data, the system can quickly evaluate the adaptability of the current segmentation strategy and automatically trigger the adjustment mechanism when necessary to cope with the challenges brought by environmental changes; through time series analysis and anomaly detection mechanisms, the changing trends of resource usage and network conditions can be more accurately evaluated to avoid waste and idleness of resources; based on real-time environmental change data and performance evaluation results, the intelligent decision-making engine can select the most appropriate segmentation strategy, optimize the allocation of computing resources, storage resources and network resources, and improve resource utilization efficiency; by predicting the changing trends of resource usage and network conditions, potential problems can be discovered in advance and corresponding preventive measures can be taken to avoid system crashes or performance degradation; when abnormal fluctuations or performance degradation occur, the system can quickly adjust the segmentation strategy. Strategy, restore the stability and reliability of the system, and ensure the smooth progress of data processing tasks; build an adjustment strategy library containing a variety of alternative segmentation strategies, providing the system with flexible selection space to adapt to different environmental changes and task requirements; with the continuous development of computing resources, storage devices and network technologies, the system can easily expand and adjust the strategy library to support more complex data processing tasks and higher performance requirements; through the intelligent decision-making engine, it can automatically select the most appropriate segmentation strategy based on real-time data and preset conditions, reduce manual intervention and decision-making time, and improve decision-making efficiency; the intelligent decision-making engine can accurately adjust the segmentation strategy according to environmental change data and performance evaluation results, ensure that the adjusted strategy can meet performance requirements to the greatest extent, and improve the accuracy of decision-making; by evaluating the effect of the adjusted segmentation strategy and feeding the evaluation results back to the real-time monitoring and adaptability evaluation links, a closed-loop optimization mechanism is formed. This mechanism can continuously iterate and optimize the performance and stability of the system, and promote the continuous improvement and development of the system.

[0129] In one embodiment of the present invention, the multi-dimensional segmentation strategy is designed based on data characteristics, and the segmentation granularity is dynamically adjusted according to real-time processing performance and data flow changes, including:

[0130] Conduct a comprehensive analysis of the preprocessed data to identify the data's intrinsic structure, distribution characteristics, correlations, and changing trends, especially focusing on those feature dimensions that have a significant impact on the segmentation effect.

[0131] Evaluate the density distribution of data in different regions or dimensions, identify sparse and dense areas, establish a real-time data flow monitoring system, and capture the dynamic changes of data flow; including the increase or decrease of data volume, the drift of data distribution, etc., to provide a basis for dynamically adjusting the segmentation granularity.

[0132] Based on the results of data characteristic analysis, the overall framework of multi-dimensional segmentation strategy is constructed, including segmentation dimension selection, segmentation granularity setting, segmentation algorithm selection, etc.; based on multiple indicators, including information gain and mutual information, the contribution of different dimensions to the segmentation effect is evaluated, and the selection of segmentation dimensions is optimized;

[0133] Through the adaptive segmentation granularity adjustment mechanism, the segmentation granularity is dynamically adjusted according to the real-time processing performance and data flow changes.

[0134] The working principle of the above technical solution is: a comprehensive and in-depth analysis of the pre-processed data is the basis for designing a multi-dimensional segmentation strategy. The analysis includes but is not limited to the intrinsic structure of the data (such as the organization of the data, whether there is a hierarchical or nested relationship), distribution characteristics (such as the distribution of data in different feature dimensions, whether there is skewness or long tail phenomenon), correlation (the degree of association between features) and change trend (the law of data change over time or conditions). Pay special attention to those feature dimensions that have a significant impact on the segmentation effect, which can often reveal important information and structure in the data. Evaluate the density distribution of data in different regions or dimensions. Identify sparse and dense areas in the data set through statistical methods or visualization techniques. The information density and processing difficulty of these areas are often different, which has an important impact on the design of the segmentation strategy. At the same time, a real-time data flow monitoring system is established to capture the dynamic changes of the data flow in real time. This includes changes such as the increase or decrease of data volume and the drift of data distribution. These changes may be caused by various factors, such as changes in data sources and adjustments to data collection frequency. The real-time monitoring system provides a key basis for dynamically adjusting the segmentation granularity. Based on the results of data characteristic analysis, the overall framework of the multi-dimensional segmentation strategy is constructed. This includes selecting the segmentation dimension, setting the segmentation granularity, and selecting the appropriate segmentation algorithm. When selecting the segmentation dimension, it is necessary to comprehensively consider multiple indicators, such as information gain, mutual information, etc., to evaluate the contribution of different dimensions to the segmentation effect. Through optimization algorithms or heuristic methods, the dimension with the most significant segmentation effect is selected as the main basis for segmentation. At the same time, according to the data characteristics and processing requirements, a reasonable segmentation granularity is set. Too large granularity may lead to reduced processing efficiency, while too small granularity may increase processing difficulty and cost. Selecting a suitable segmentation algorithm is also an important part of building a multi-dimensional segmentation strategy. The algorithm should be selected based on the characteristics of the data and processing requirements. Through the adaptive segmentation granularity adjustment mechanism, the segmentation granularity is dynamically adjusted according to the real-time processing performance and data flow changes. This mechanism analyzes whether the current processing performance meets the requirements and whether the data flow has changed significantly by monitoring the feedback data of the system in real time. If there is a bottleneck in the processing performance or the data flow changes significantly, the adjustment of the segmentation granularity is triggered. The adjustment process can be automatic or carried out under manual intervention. By dynamically adjusting the segmentation granularity, it can be ensured that the system can maintain efficient and stable operation under different loads and data flow changes.

[0135] The effect of the above technical solution is: by comprehensively analyzing the intrinsic structure, distribution characteristics, correlation and change trend of the data, and paying special attention to the characteristic dimensions that have a significant impact on the segmentation effect, a more accurate and efficient multi-dimensional segmentation strategy can be designed. This strategy can more effectively divide data, reduce redundancy and duplication in data processing, and thus improve the overall processing efficiency. Evaluating the density distribution of data in different regions or dimensions and setting the segmentation granularity accordingly can ensure the reasonable allocation of resources during the processing process. A larger segmentation granularity is used in sparse areas to reduce processing overhead, and a smaller segmentation granularity is used in dense areas to ensure processing accuracy. This strategy can optimize resource utilization and avoid waste or excessive consumption of resources. A real-time data flow monitoring system is established to capture the dynamic changes of data flows in real time, including the increase and decrease of data volume, the drift of data distribution, etc. This provides a key basis for dynamically adjusting the segmentation granularity, so that the system can automatically adjust the strategy according to real-time processing performance and data flow changes, and maintain an efficient and stable operating state. This adaptive capability enhances the flexibility and adaptability of the system, enabling it to better cope with complex and changing data processing scenarios. Based on multiple indicators (such as information gain, mutual information, etc.), the contribution of different dimensions to the segmentation effect is evaluated, and the selection of segmentation dimensions is optimized to ensure the scientificity and rationality of the segmentation strategy. This strategy can more accurately identify important information and structures in the data, thereby improving the segmentation effect and providing more accurate and useful data support for subsequent data analysis and mining. By dynamically adjusting the segmentation granularity, the processing cost can be reduced while ensuring the processing effect. In the case of limited resources, through reasonable segmentation strategy design and dynamic adjustment mechanism, the consumption and waste of resources can be reduced without sacrificing the processing effect, thereby reducing the overall processing cost. For systems or applications that need to process large amounts of data, efficient and stable data processing is one of the important factors to improve user experience. The above technical solution can significantly improve the overall performance and stability of the system by improving data processing efficiency, optimizing resource utilization, and enhancing system adaptability, thereby improving user experience.

[0136] In one embodiment of the present invention, the S2 includes:

[0137] Analyze the logical correlation between each data subset, including causal relationship, time series dependency, spatial proximity or other related features. Determine the dependencies between subtasks based on the analysis results, which subtask results need to be used as input for the next subtask, and construct a task dependency graph;

[0138] Based on the identified dependencies, the order of task execution is designed to form a linear task chain or a complex directed acyclic graph (DAG) structure;

[0139] Formulate task scheduling strategies for task chain execution, including priority scheduling, FIFO (first in, first out), LIFO (last in, first out) or other custom strategies. Encapsulate subtasks into executable units and configure input data and output paths; design task transmission mechanisms to ensure that the intermediate results generated after the completion of the previous subtask can be safely and timely transferred to the subsequent subtasks.

[0140] Implement a task execution monitoring system to track the execution status of the task chain, including the completion of each subtask, resource usage, and possible exceptions. Establish a task coordination mechanism to reschedule or adjust the order of tasks when necessary to cope with changing needs or emergencies.

[0141] When all subtasks are completed in sequence, their respective results are integrated to form a complete task chain.

[0142] The working principle of the above technical solution is as follows: First, a comprehensive analysis of the logical correlation between data subsets is conducted, which may involve causal relationships, time series dependencies, spatial proximity or other relevant features, so as to determine the data dependencies between different subtasks. Based on these analysis results, a task dependency graph is constructed to clarify which subtask output needs to be used as the input of another subtask, thereby forming a series of interrelated task chains. An ordered task chain structure is designed based on the dependencies between subtasks, which may be a simple linear sequence or a complex directed acyclic graph to reflect the nonlinear front-to-back execution requirements between tasks. A task scheduling strategy is formulated, and priority scheduling, first-in-first-out, last-in-first-out or other custom strategies are selected according to factors such as the importance and timeliness of the task to ensure that the tasks are executed in a reasonable order. The subtasks are encapsulated into executable units, and the input data and output paths required for each subtask are configured to ensure that the data flow can flow smoothly between tasks. A task execution monitoring system is implemented to track the execution status of each subtask in real time, including completion status, resource occupancy status, and any possible abnormal situations. Design a task coordination mechanism that can dynamically adjust the task execution plan according to the actual situation during the execution process. For example, when there is a task delay, resource shortage or other changes, the task can be rescheduled or the task execution order can be adjusted in a timely manner. Ensure that during the task execution process, the intermediate results generated by the previous subtask can be safely and promptly passed to the subsequent subtasks. Usually, message queues, shared storage or other distributed data exchange technologies are used to achieve real-time and reliable transmission. When all subtasks are completed in the set order, the final results generated by each subtask are integrated to form a complete output of the entire task chain.

[0143] The effect of the above technical solution is: by analyzing the logical correlation between data subsets, the dependency relationship between subtasks is clarified, and a task dependency graph is constructed, thereby forming a reasonable and orderly task chain. This can ensure that subtasks are executed on demand and in an orderly manner, avoid invalid waiting and repeated calculations, and greatly improve the efficiency of the entire task execution; according to task dependencies and business needs, flexible task scheduling strategies (such as priority scheduling, FIFO or LIFO, etc.) are adopted to ensure that important or urgent tasks are executed first and computing resources are effectively utilized. At the same time, through task encapsulation and input / output path configuration, the consistency of task execution and the security of data transmission are guaranteed; through the task execution monitoring system, the execution status of each subtask can be grasped in real time, including progress, resource usage and abnormal conditions, which helps to find problems and take measures in time. The established task coordination mechanism can quickly adjust the order of task execution when demand changes or emergencies occur, ensuring the robustness and high availability of the system. By splitting complex large tasks into multiple independent and manageable subtasks, an intuitive and easy-to-understand task chain structure is formed. Whether it is a linear task chain or a complex directed acyclic graph structure, the logical hierarchy of the entire system is clear, which is convenient for development, debugging and maintenance. Task execution is arranged according to the relevance of data and the order of processing, ensuring that the intermediate results can be accurately passed to the downstream dependent subtasks, and finally integrating the results of all subtasks to form a complete task chain output, ensuring the correctness and integrity of data processing. This technical solution can easily cope with the challenges of large-scale data processing, allowing the system to elastically scale according to changes in data volume and computing resources, while also leaving ample room for future functional expansion. At the same time, the fault tolerance and self-healing capabilities of the task chain have been enhanced. Even when problems occur in some subtasks, they can be quickly restored through task scheduling and coordination mechanisms to maintain the stable operation of the overall system.

[0144] In one embodiment of the present invention, S3 includes:

[0145] Initialize the distributed computing environment, deploy and start the task execution framework, which includes Hadoop and Spark; load each subtask in the task chain into the computing resource pool;

[0146] Start each subtask to process its corresponding data subset in parallel, and use multi-core CPU, GPU or other parallel computing resources to accelerate the data aggregation process. Each subtask is executed independently on its computing node and generates the corresponding subset aggregation results;

[0147] A resource monitoring module is set up to continuously monitor the computing resources occupied by each subtask during execution, including key indicators such as CPU usage, memory usage, and network bandwidth; and the resource usage data is reported to the resource scheduling center; the occupied computing resources are calculated using the following formula:

[0148]

[0149] Among them, R represents a certain measure of resource occupancy or task load score, which reflects the overall occupancy of subtasks on computing resources. S represents the total number of subtasks in the system. Represents the computational complexity, required computing power, workload, or other important measurement factors related to task execution of the j-th subtask. Indicates the amount of computing resources (such as memory, CPU time, etc.) used by the j-th subtask. It represents the standard deviation and is used to normalize the resource usage. T represents the total time of task execution.

[0150] Based on the collected resource usage, a load balancing algorithm is used to evaluate whether the current system resource allocation is balanced; the load balancing algorithm includes but is not limited to strategies such as polling, minimum load priority, and maximum idle resource priority.

[0151] The resource scheduler dynamically adjusts the computing resource allocation of each subtask based on the evaluation results of the load balancing algorithm;

[0152] For subtasks with excessive resource usage, balance can be achieved by reducing their resource shares or migrating some tasks to computing nodes with sufficient resources.

[0153] For nodes with low resource utilization, the number of subtasks or the amount of resources they carry can be appropriately increased.

[0154] After the parallel processing of the subtasks is completed, the summary results of each subtask are integrated together through the synchronization mechanism to form a preliminary summary of the entire original data set;

[0155] And according to the resource scheduling effect in the actual execution process, continuous feedback is provided to optimize the resource scheduling strategy.

[0156] The working principle of the above technical solution is as follows: First, build and initialize an environment that supports distributed computing, and deploy corresponding task execution frameworks (such as Hadoop or Spark). These frameworks can effectively manage distributed computing tasks and coordinate the resources and task execution of each computing node. Each subtask in the task chain is loaded into the computing resource pool and prepared for parallel execution. Start each subtask to independently and parallelly process its data subset on the corresponding computing node, and use multi-core CPU, GPU or other parallel computing resources to accelerate the data aggregation process. Each subtask generates a summary result of its corresponding data subset. During the task execution process, a resource monitoring module is set to monitor the computing resources used by each subtask in real time, including but not limited to key indicators such as CPU usage, memory usage, network bandwidth, etc., and report the monitoring data to the resource scheduling center. According to the collected resource usage, a load balancing algorithm (such as polling, minimum load priority, maximum idle resource priority, etc.) is used to evaluate whether the system resource allocation is balanced. The resource scheduler dynamically adjusts the computing resource allocation of each subtask based on the evaluation results of the load balancing algorithm. For subtasks with excessive resource usage and high pressure, the purpose of balancing is achieved by reducing their resource share or migrating some tasks to computing nodes with more abundant resources. For nodes with low resource utilization, the number of subtasks or the amount of resources are appropriately increased to make full use of idle resources. After all subtasks are processed in parallel, the summary results of each subtask are integrated with the help of the synchronization mechanism to form a preliminary summary result of the entire original data set. According to the resource scheduling effect in the actual execution process, the resource scheduling strategy is continuously fed back to optimize, in order to achieve better resource utilization efficiency and faster processing speed in future task execution.

[0157] The effect of the above technical solution is: by initializing the distributed computing environment and deploying task execution frameworks such as Hadoop or Spark, the subtasks in the task chain can be loaded into the computing resource pool to achieve parallel processing. The use of parallel computing resources such as multi-core CPUs and GPUs significantly accelerates the data aggregation process and greatly improves the processing efficiency. Setting a resource monitoring module to monitor the resource occupancy during the execution of subtasks in real time, including core indicators such as CPU usage, memory usage, and network bandwidth, can effectively grasp the real-time status of system resources and report this information to the resource scheduling center. The load balancing algorithm is used to dynamically adjust resource allocation, such as polling, minimum load priority, maximum idle resource priority, etc., to ensure the balanced use of system resources. For subtasks that occupy too many resources, their resource shares can be appropriately reduced or migrated to computing nodes with sufficient resources, while for nodes with low resource utilization, subtasks or resource quotas can be increased to achieve optimal resource configuration. By real-time monitoring and dynamic adjustment of resources, system instability caused by overload of computing nodes can be effectively prevented, the reliability of task execution can be improved, and all subtasks can be ensured to be successfully completed. This technical solution enables the system to have good scalability, and can flexibly increase or decrease subtasks and allocate resources according to actual needs and resource conditions, which is convenient for later maintenance and optimization of the system. After the parallel processing of subtasks is completed, the summary results of each subtask are integrated through a synchronization mechanism to form a preliminary summary of the entire original data set. This parallel summary method greatly shortens the data processing time and improves the timeliness of data processing. According to the resource scheduling effect during the actual execution process, the resource scheduling strategy is continuously fed back and optimized to ensure that the system is always in the best operating state and continuously improve the overall data processing performance. Through the above formula, the system can quantify the occupancy of each subtask in terms of computing resources, providing an objective and unified standard for resource scheduling. In the formula, and The combination of takes into account the computational complexity of the subtasks and the amount of resources they actually occupy, thereby truly reflecting the degree of influence of each subtask on the overall resource pool; based on the calculated resource occupancy rate, the resource scheduler can more accurately evaluate whether the current system resource allocation is balanced. When it is found that some subtasks occupy too much or too little resources, the scheduler can dynamically adjust the resources according to the load balancing algorithm to ensure that the overall system resources are fully utilized and avoid resource waste or resource competition; by continuously monitoring the resource usage of each subtask during execution and adjusting resource allocation according to the calculation results of the formula, the entire system can achieve adaptive optimization scheduling. This means that in the actual operation process, the system can automatically adjust the strategy according to the changes in task load, and always maintain a high resource utilization rate and task execution efficiency; the time parameter T in the formula indicates that the resource occupancy is sampled in units of time windows, which enables the system to monitor resource usage in real time and update the scheduling strategy according to the latest data to ensure that the system can respond quickly to changes in task load.

[0158] In one embodiment of the present invention, the S34 includes:

[0159] Collect multi-dimensional resource usage data, including CPU, memory usage, network bandwidth usage, disk I / O, and cache hit rate; based on the collected multi-dimensional resource usage data, identify performance bottlenecks in the current system through machine learning, such as CPU overload, memory leak, network congestion, etc.

[0160] Construct a load pressure assessment model to assess the load pressure of each computing node based on multiple factors, including the execution progress of subtasks, resource consumption rate, remaining workload, etc.

[0161] Through load balancing algorithms based on multi-objective optimization, such as multi-objective genetic algorithms and particle swarm optimization algorithms, the dual goals of efficient utilization of computing resources and load balancing are achieved; combined with the resource utilization evaluation results, the parameters and strategies in the load balancing algorithm are dynamically adjusted, such as adjusting the task migration threshold, optimizing the task allocation weight, etc., to adapt to the real-time changes in the system status.

[0162] The resource scheduler generates resource scheduling decisions, including task migration and resource reallocation, based on the evaluation results and dynamic adjustment strategies of the load balancing algorithm; sends the generated resource scheduling decisions to each computing node to perform the corresponding task migration and resource reallocation operations;

[0163] By monitoring the system status and resource usage after resource scheduling, the scheduling effect is evaluated, including indicators such as load balancing, processing efficiency improvement, and resource utilization; based on the evaluation results, the resource scheduling strategy is continuously optimized, including adjusting the parameters of the load balancing algorithm, improving the resource usage evaluation model, etc., to form a closed-loop optimization mechanism.

[0164] The working principle of the above technical solution is as follows: First, the system needs to comprehensively collect usage data of various key resources in the computing environment, including but not limited to CPU usage, memory usage, network bandwidth occupancy, disk I / O performance, and cache hit rate. These data are the basis for evaluating system performance and identifying potential bottlenecks. Based on the collected multi-dimensional resource usage data, the system uses machine learning technology for analysis and modeling. The machine learning model can identify patterns and anomalies in the data, thereby helping the system locate the current performance bottleneck. These bottlenecks may include CPU overload, memory leaks, network congestion, disk I / O bottlenecks, etc. After identifying the bottleneck, the system can take targeted optimization measures. In order to more accurately evaluate the load pressure of each computing node, the system constructs a load pressure evaluation model. The model comprehensively considers multiple factors, such as the execution progress of subtasks, resource consumption rate, remaining workload, etc. Through comprehensive analysis of these factors, the model can give the load pressure evaluation results of each node, providing an important reference for subsequent load balancing and resource scheduling. The system uses a load balancing algorithm based on multi-objective optimization (such as a multi-objective genetic algorithm, a particle swarm optimization algorithm, etc.) to make resource scheduling decisions. These algorithms are designed to achieve the dual goals of efficient utilization of computing resources and load balancing. During the algorithm operation, the system will dynamically adjust the algorithm parameters and strategies according to the resource usage evaluation results, such as adjusting the task migration threshold and optimizing the task allocation weight, so as to adapt to the real-time changes in the system status. The resource scheduler generates specific resource scheduling decisions based on the evaluation results and dynamic adjustment strategies of the load balancing algorithm. These decisions may include operations such as task migration and resource reallocation. The scheduler sends the generated decisions to each computing node, and the node performs the corresponding task migration and resource reallocation operations according to the received instructions. Finally, the system evaluates the scheduling effect by monitoring the system status and resource usage after resource scheduling. The evaluation indicators include load balancing, processing efficiency improvement, resource utilization, etc. Based on the evaluation results, the system continuously feeds back to optimize the resource scheduling strategy. This may include adjusting the load balancing algorithm parameters and improving the resource usage evaluation model. Through such a closed-loop optimization mechanism, the system can continuously optimize its resource scheduling performance to ensure efficient utilization of computing resources and stable operation of the system.

[0165] The effect of the above technical solution is: through the collection and machine learning analysis of multi-dimensional resource usage data, the system can identify the performance bottlenecks in the current system in real time and accurately, such as CPU overload, memory leak, network congestion, etc. This real-time performance monitoring and bottleneck identification capability enables the system to respond quickly and take corresponding optimization measures, thereby avoiding the impact of performance problems on the overall business. The constructed load pressure evaluation model is based on a comprehensive consideration of multiple factors, including the execution progress of subtasks, resource consumption rate, remaining workload, etc., and can more accurately evaluate the load pressure of each computing node. This accurate evaluation provides reliable data support for subsequent load balancing and resource scheduling, and helps to achieve more reasonable resource allocation and scheduling. The use of load balancing algorithms based on multi-objective optimization, such as multi-objective genetic algorithms, particle swarm optimization algorithms, etc., can simultaneously consider the dual goals of efficient utilization of computing resources and load balancing. By dynamically adjusting the parameters and strategies in the algorithm, such as adjusting the task migration threshold and optimizing the task allocation weight, the system can more flexibly respond to real-time changes in system status and achieve more efficient and balanced load distribution. The resource scheduler automatically generates resource scheduling decisions based on the evaluation results and dynamic adjustment strategies of the load balancing algorithm, and sends them to each computing node for execution. This automated resource scheduling mechanism reduces the need for manual intervention and improves the efficiency and accuracy of resource scheduling. At the same time, through real-time task migration and resource reallocation operations, the system can respond to load changes more quickly and ensure the stable operation of the system. By monitoring the system status and resource usage after resource scheduling, the system can evaluate the scheduling effect and continuously provide feedback to optimize the resource scheduling strategy based on the evaluation results. This closed-loop optimization mechanism enables the system to continuously improve its performance, including improving load balancing, processing efficiency, resource utilization and other indicators. By adjusting the load balancing algorithm parameters, improving the resource usage evaluation model and other measures, the system can continuously adapt to new business needs and system status changes.

[0166] In one embodiment of the present invention, the S4 includes:

[0167] S41. Select the merge algorithm based on the data type and specific requirements of the summary task. For example, for simple summaries such as sum and average, you can directly add or average; for more complex statistical tasks, you may need to use methods such as merge sort and distributed reduce operations. And clarify the specific implementation details of the merge algorithm; including how to compare and merge the intermediate results from different subtasks to ensure the consistency and accuracy of the data during the merge process.

[0168] S42, create a data structure for storing global summary results, the data structure includes arrays, queues, heaps, etc., initialized according to data characteristics and requirements of the merging algorithm. After completing the processing of all subtasks, collect the summary results of each subset from each computing node and transmit them to the master node or the designated merging node;

[0169] S43, executing a merging algorithm, and gradually integrating the summary results of each subset according to a preset merging rule to form a complete global summary result;

[0170] S44, performing consistency check on the merged global summary results. If there is a loss of accuracy or a large deviation, fine-tune and optimize the merging algorithm and perform a secondary merge.

[0171] S45. Collect feedback information based on the performance and result quality during the merging process, and continuously optimize the merging algorithm and data processing flow based on the feedback information.

[0172] The working principle of the above technical solution is as follows: according to the data type and the specific requirements of the summary task, select a suitable merging algorithm. For example, simple summation and average values ​​can be merged by accumulation or averaging. Complex statistical tasks may require the use of merge sort, distributed reduce and other operations to clarify the specific implementation details of the merging algorithm, so as to ensure that the intermediate results generated by different subtasks can be correctly and accurately compared and merged to ensure the consistency and accuracy of the data during the merging process. Create a data structure suitable for storing global summary results, such as arrays, queues, heaps, etc., and initialize them according to the characteristics of the data and the requirements of the selected merging algorithm to prepare for subsequent merging operations. When all subtasks are processed in parallel, collect the summary results of each subset from each computing node, and transmit these results to the master node or a node dedicated to merging results. Execute the preset merging algorithm, and gradually integrate the summary results of each subset according to the merging rules to form a global summary result of the entire original data set. Perform a consistency check on the merged global summary result to check whether there is a loss of precision or excessive deviation. If there is a problem, fine-tune and optimize the merging algorithm, and perform necessary secondary merging to ensure the accuracy of the result. Based on the performance (such as execution time, resource consumption, etc.) and result quality (such as accuracy, error range, etc.) of the merging process, feedback information is collected to continuously optimize the merging algorithm and the overall data processing process, thereby continuously improving the system processing capabilities and efficiency.

[0173] The effect of the above technical solution is: select an appropriate merging algorithm according to the characteristics of the data type and the summary task, such as simple summation and average value can be directly accumulated or averaged, and complex statistical tasks can use merge sort, distributed reduce and other operations, which can efficiently and accurately integrate the summary results of subtasks to obtain global summary results. Clarify the specific implementation details of the merging algorithm to ensure that the consistency and accuracy of the data can be maintained when comparing and merging the intermediate results from different subtasks, and avoid distortion of the summary results due to errors in the data merging process. According to the characteristics of the data and the requirements of the merging algorithm, create a suitable global summary result data structure, such as arrays, queues, heaps, etc., so that the data integration process is more flexible and efficient, and meets the data storage and processing requirements in different scenarios. After completing the processing of all subtasks, the summary results of the subsets can be quickly collected from each computing node and transmitted to the master node or the designated merging node, realizing efficient data integration and transmission, and reducing the delay caused by network transmission. Perform consistency check on the merged global summary results. If there is a loss of accuracy or too large deviation, it can be fine-tuned and optimized in time, and a secondary merge can be performed to ensure the quality of the final summary result. Collect feedback information based on the performance and result quality during the merging process, and continuously optimize the merging algorithm and the entire data processing process, so that the system can self-learn and self-adapt, and continuously improve processing efficiency and result accuracy.

[0174] One embodiment of the present invention, as Figure 2 As shown, a massive data aggregation system based on task chain and divide-and-conquer method, the system includes:

[0175] Data partitioning module: divides the massive original data set to be processed into multiple independent data subsets according to the preset partitioning rules, each of which represents a subtask;

[0176] Task chain forming module: based on the relevance of data and the processing order, the subtasks are connected in order to form a task chain;

[0177] Data aggregation module: each subtask in the task chain is processed in parallel to obtain the aggregation result of the data subset;

[0178] Result integration module: integrates the summary results of data subsets through a merging algorithm to obtain the global summary results of the entire data set.

[0179] The working principle of the above technical solution is as follows: First, according to the preset division rules, the massive original data set to be processed is intelligently divided into multiple independent and related data subsets. Each data subset represents an independent processing unit, namely a subtask. Such a divide-and-conquer strategy greatly reduces the pressure on a single computing node, and also conforms to the concept of distributed computing, making big data processing more efficient and feasible; based on the correlation and processing order between data subsets, each subtask is connected in series according to a logical relationship to form one or more task chains. The construction of the task chain fully considers the continuity and dependency of data processing, and ensures the correctness and integrity of data processing. The output of each subtask will be used as the input of the next subtask, forming an orderly, pipelined processing flow; each subtask in the task chain is processed in a parallel computing manner. Through effective resource scheduling strategies, such as load balancing algorithms, each subtask is reasonably allocated to different computing resources to achieve concurrent execution, which greatly improves the data processing speed. During the parallel processing, the computing resource load of each subtask is monitored in real time, and resource allocation is adjusted dynamically to ensure the optimal overall operating efficiency of the system; when all subtasks complete their respective processing tasks, the summary results of each data subset are obtained. Next, the summary results of each subset are integrated through a pre-selected merging algorithm (such as distributed Reduce operation, merge sort, etc.). This merging process can ensure that the global summary results of the entire data set are accurately restored from a large number of scattered local summary results, thereby meeting business needs and data analysis purposes.

[0180] The effect of the above technical solution is: by dividing the original data set into multiple independent data subsets and processing each subtask in parallel, the data processing speed is significantly improved, and the problem of huge data volume in the big data environment is effectively dealt with. This method gives full play to the advantages of distributed computing and parallel processing, so that the summary work that may take a long time to complete can be completed quickly; based on the task chain and divide-and-conquer strategy, the task allocation and resource scheduling can be dynamically adjusted according to the size of the data subset and the computing resource status to achieve load balancing. This can avoid overloading of a single computing node and make full use of the overall computing power of the cluster, thereby improving the system throughput and response speed; by building a task chain based on the relevance and processing order of the data, it is ensured that the correct data dependency can be followed even in the process of parallel processing, ensuring the accuracy and consistency of the summary results; the divide-and-conquer method makes the processing process have good scalability. When the amount of data increases or the computing resources increase, only more subtasks and computing nodes need to be added without making major changes to the entire processing flow. At the same time, since each subtask is relatively independent, fault recovery and system maintenance are also more convenient; compared with processing the entire original data set at one time, the divide-and-conquer and parallel processing methods can save computing resources, shorten processing time, and indirectly reduce hardware investment costs and energy consumption, which has a positive impact on long-term operations and green energy conservation; through the pre-set merging algorithm, the summary results of each data subset can be accurately merged into the global summary result of the entire data set, ensuring that the summary result fully reflects the information of the original data set.

[0181] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for summarizing massive data based on task chain and divide-and-conquer method, characterized in that: The method comprises: S1. According to the preset division rules, the massive original data set to be processed is divided into multiple independent data subsets, each data subset represents a subtask; S2. Based on the relevance of the data and the processing order, the subtasks are connected in order to form a task chain; S3, performing parallel processing on each subtask in the task chain to obtain a summary result of the data subset; S4, integrating the summary results of the data subsets through a merging algorithm to obtain a global summary result of the entire data set; The S4 comprises: S41. Select the merging algorithm based on the data type and specific requirements of the aggregation task, and clarify the specific implementation details of the merging algorithm; S42, creating a data structure for storing global summary results, and after completing the processing of all subtasks, collecting the summary results of each subset from each computing node, and transmitting them to the master control node or the designated merge node; S43, executing a merging algorithm, and gradually integrating the summary results of each subset according to a preset merging rule to form a complete global summary result; S44, performing consistency check on the merged global summary results. If there is a loss of accuracy or a large deviation, fine-tune and optimize the merging algorithm and perform a secondary merge. S45. Collect feedback information based on the performance and result quality of the merging process, and continuously optimize the merging algorithm and data processing flow based on the feedback information; Said S1 comprises: S11, dividing the massive original data set into multiple partitions, preprocessing the data in each partition, and after the preprocessing is completed, aggregating the data in each partition to form a complete data set after preprocessing; S12, according to data characteristics and computing resources, a divide-and-conquer rule is preset, and according to the preset divide-and-conquer rule, a data segmentation operation is performed to divide the original data set into multiple independent data subsets; S13, defining each generated data subset as an independent subtask, wherein each subtask is executed as a separate work unit in the distributed computing environment; The S12 comprises: Use statistical methods and machine learning algorithms to extract features from massive raw data sets and identify key dimensions and outliers in the data; Analyze the dependencies between elements in the data set, and evaluate the resource requirements and potential performance bottlenecks of different segmentation strategies based on currently available computing resources and network conditions; Design a multi-dimensional segmentation strategy based on data characteristics, and dynamically adjust the segmentation granularity according to real-time processing performance and data flow changes; Use heuristic algorithms or reinforcement learning algorithms to predict and optimize the load distribution of the segmented data subsets in a distributed environment; Before data segmentation, parallel computing is used to preprocess the data; based on the preset divide-and-conquer rules and the data characteristics of real-time analysis, the intelligent segmentation algorithm is used to segment the preprocessed data; Create independent packaging files for each generated data subset and generate indexing mechanisms; Evaluate the processing efficiency, resource utilization, and independence of subtasks of the segmented data subsets in a distributed computing environment through simulation or small-scale testing; According to the evaluation results, the divide-and-conquer rules and segmentation strategies are iteratively optimized, and the optimized divide-and-conquer rules and segmentation strategies are incorporated into the knowledge base.

2. According to claim 1, a method for summarizing massive data based on task chain and divide-and-conquer method, characterized in that: The dependencies between the elements in the data set are analyzed, and the resource requirements and potential performance bottlenecks of different segmentation strategies are evaluated in combination with the currently available computing resources and network conditions, including: Uniquely identify and label each element in a massive data set, and use causal reasoning algorithms to analyze the direct and indirect dependencies between elements and build a data dependency graph; Quantitatively evaluate the identified dependencies, build a resource usage model based on the currently available computing resources and network conditions, and simulate the resource requirements of different data processing tasks; Based on the data dependency graph, multiple data segmentation strategies are preset. The resource usage model is used to predict the execution performance of each segmentation strategy under different resource configurations and identify potential performance bottlenecks. Monitor the dynamic changes of computing resources and network conditions in real time, dynamically evaluate the adaptability and effectiveness of the current segmentation strategy based on environmental change data, and automatically trigger the adjustment of the segmentation strategy according to the evaluation results through the feedback and adjustment mechanism.

3. According to claim 2, a method for summarizing massive data based on task chain and divide-and-conquer method, characterized in that: The real-time monitoring of the dynamic changes of computing resources and network conditions, based on the environmental change data, dynamically evaluating the adaptability and effectiveness of the current segmentation strategy, and automatically triggering the adjustment of the segmentation strategy according to the evaluation results through the feedback and adjustment mechanism, include: Through the real-time monitoring system, the computing resources and network status data are collected in real time; and the collected raw data is pre-processed; Use time series analysis to analyze historical data on computing resources and network conditions, predict changing trends, and use anomaly detection mechanisms to monitor abnormal fluctuations in resource usage or network conditions; Compare and analyze the environmental change data with the preset performance indicators to evaluate the adaptability of the current segmentation strategy in different environments; According to the results of adaptability analysis, set the strategy that triggers the segmentation strategy adjustment, and build an adjustment strategy library containing multiple alternative segmentation strategies; Through the intelligent decision-making engine, the segmentation strategy is selected from the adjustment strategy library for adjustment according to the real-time environmental change data and preset trigger conditions; The effectiveness of the adjusted segmentation strategy is evaluated and the evaluation results are fed back to the real-time monitoring and adaptive evaluation links to form a closed-loop optimization mechanism.

4. According to claim 1, a method for summarizing massive data based on task chain and divide-and-conquer method, characterized in that: The multi-dimensional segmentation strategy is designed based on data characteristics, and the segmentation granularity is dynamically adjusted according to real-time processing performance and data flow changes, including: Evaluate the density distribution of data in different regions or dimensions, identify sparse and dense areas, establish a real-time data flow monitoring system, and capture the dynamic changes of data flow; Based on the results of data characteristic analysis, the overall framework of multi-dimensional segmentation strategy is constructed to evaluate the contribution of different dimensions to the segmentation effect and optimize the selection of segmentation dimensions; Through the adaptive segmentation granularity adjustment mechanism, the segmentation granularity is dynamically adjusted according to the real-time processing performance and data flow changes.

5. The method for summarizing massive data based on task chain and divide-and-conquer method according to claim 1, characterized in that: The S2 comprises: S21, analyzing the logical correlation between each data subset, determining the front-to-back dependency relationship between subtasks according to the analysis results, and constructing a task dependency graph; S22. Design the order of task execution based on the identified dependencies and form a linear task chain or a complex directed acyclic graph structure; S23, formulate a task scheduling strategy for task chain execution, encapsulate subtasks into executable units, and configure input data and output paths; S24. Implement a task execution monitoring system to track the execution status of the task chain and establish a task coordination mechanism; S25. When all subtasks are completed in sequence, their respective results are integrated to form a complete task chain.

6. The method for summarizing massive data based on task chain and divide-and-conquer method according to claim 1, characterized in that: The S3 includes: S31, initialize the distributed computing environment, deploy and start the task execution framework, and load each subtask in the task chain into the computing resource pool; S32, start each subtask to process its corresponding data subset in parallel, each subtask is executed independently on its computing node, and generates a corresponding subset summary result; S33, setting a resource monitoring module to continuously monitor the computing resources occupied by each subtask during execution, and reporting resource usage data to the resource scheduling center; S34, based on the collected resource usage, a load balancing algorithm is used to evaluate whether the current system resource allocation is balanced; the resource scheduler dynamically adjusts the computing resource allocation of each subtask based on the evaluation result of the load balancing algorithm; S35, after the parallel processing of the subtasks is completed, the summary results of each subtask are integrated together through the synchronization mechanism to form a preliminary summary of the entire original data set; S36. Based on the resource scheduling effect during the actual execution process, continuously provide feedback to optimize the resource scheduling strategy.

7. The method for summarizing massive data based on task chain and divide-and-conquer method according to claim 6, characterized in that: The S34 comprises: Collect multi-dimensional resource usage data, and use machine learning to identify performance bottlenecks in the current system based on the collected multi-dimensional resource usage data; Construct a load pressure assessment model to evaluate the load pressure of each computing node based on multiple factors; Through the load balancing algorithm based on multi-objective optimization and combined with the resource usage evaluation results, the parameters and strategies in the load balancing algorithm are dynamically adjusted; The resource scheduler generates resource scheduling decisions based on the evaluation results of the load balancing algorithm and the dynamic adjustment strategy, and sends the generated resource scheduling decisions to each computing node to perform corresponding task migration and resource reallocation operations; By monitoring the system status and resource usage after resource scheduling, the scheduling effect is evaluated, and based on the evaluation results, continuous feedback is provided to optimize the resource scheduling strategy.

8. A system for implementing the method for summarizing massive data based on task chain and divide-and-conquer method as claimed in claim 1, characterized in that: The system comprises: Data partitioning module: divides the massive original data set to be processed into multiple independent data subsets according to the preset partitioning rules, each of which represents a subtask; Task chain forming module: based on the relevance of data and the processing order, the subtasks are connected in order to form a task chain; Data aggregation module: each subtask in the task chain is processed in parallel to obtain the aggregation result of the data subset; Result integration module: integrates the summary results of data subsets through a merging algorithm to obtain the global summary results of the entire data set.