Big data distributed storage and parallel processing cooperation method based on cloud computing

Abstract

The invention discloses a big data distributed storage and parallel processing collaboration method based on cloud computing. The method comprises the following steps: sensing data stream characteristics in real time through a self-adaptive dynamic partitioning engine, dynamically adjusting a partitioning strategy and generating a metadata label; constructing a node selection model through a comprehensive evaluation algorithm, selecting storage nodes to form an optimal storage cluster, and dynamically adjusting a resource matching weight coefficient based on a load state through a load optimization module; decomposing a data processing task into parallel subtask units, and constructing a dual-objective optimization model; triggering a dynamic rebalance mechanism through a distributed monitoring agent in combination with a hierarchical early warning strategy; and constructing a multi-level cache system to optimize a data access path, outputting a final result, pushing the final result to the user terminal, and updating the knowledge base. Through collaborative optimization of dynamic partitioning, multi-dimensional resource scheduling, elastic scaling and intelligent caching technologies, the resource utilization rate, the load balancing capacity and the stability in a high-concurrency scene of the system are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of cloud computing and big data processing, and discloses a collaborative method for distributed storage and parallel processing of big data based on cloud computing. Background Art

[0002] With the deep penetration of big data applications in fields such as the Internet, Internet of Things, and finance, the data scale and processing complexity in the cloud computing environment continue to climb. Distributed systems need to cope with multiple challenges such as high-concurrency access, real-time processing of heterogeneous data streams, and dynamic resource scheduling.

[0003] Currently, the mainstream distributed storage and computing architectures still follow the design paradigm based on static rules: data storage mostly adopts a fixed block strategy, task scheduling relies on a load balancing mechanism triggered by simple thresholds, and elastic scaling is achieved by manually presetting resource thresholds for expansion or contraction. Such methods can still meet the basic requirements in scenarios with controllable scale and stable load in the early stage. However, with the diversification of data types and the dynamism of business scenarios, the limitations of traditional technologies are gradually exposed. Especially in the data block division link, the fixed-size block division rule is difficult to adapt to the characteristics of data streams, resulting in fragmented and redundant storage space and increasing I / O overhead; the load distribution algorithm only focuses on a single index and lacks collaborative analysis of multi-dimensional resource states such as CPU, memory, and disk bandwidth, easily leading to problems such as local node overload and low global resource utilization. In addition, the elastic scaling mechanism, due to relying on manual experience to set thresholds, is difficult to accurately predict the load fluctuation trend, and the lag in expansion response often leads to service degradation or interruption in high-concurrency scenarios; when migrating resources across regions, traditional solutions do not fully consider factors such as network latency and data consistency maintenance costs, and the migration decision-making has a high degree of blindness, which may exacerbate system performance jitter. At the same time, the static assumptions about the business access mode in the data caching and parallel processing links result in frequent problems such as ineffective caching of frequently accessed data and blocking of the data processing pipeline, further restricting the system throughput and real-time performance. The above defects together lead to core bottlenecks such as resource waste, response delay, and insufficient scalability in existing systems under dynamic load, and it is difficult to meet the requirements of the industrial community for a new generation of cloud computing platforms with high elasticity, low latency, and high consistency. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems in cloud computing systems caused by the rigidity of the static block strategy, the inefficiency of the single-dimensional load balancing mechanism, the lag in elastic scaling response, and the high overhead of data processing, resulting in waste of storage resources, imbalance of node loads, response delay for burst traffic, contradiction between data consistency guarantee and real-time performance, and insufficient dynamic adaptability of the system, seriously restricting the storage efficiency, processing performance, and business continuity in large-scale high-concurrency scenarios.

[0005] To solve the above technical problems, the technical solution provided by the present invention is a collaborative method for big data distributed storage and parallel processing based on cloud computing, including the following steps: Step 1: Receive the original data stream, divide the original data stream into data blocks through an adaptive dynamic chunking engine, dynamically calculate the size of the divided data blocks and perform verification, and generate metadata tags. Step 1.1: Extract the data features of the original data stream through a data stream feature extraction module. Step 1.2: The chunking rule generator performs data block chunking according to the data features obtained in Step 1.1 by using an adaptive chunking strategy. The adaptive chunking strategy includes: 1) Use a fixed-size chunking method to process data with entropy value changes exceeding the threshold. 2) Record boundary chunking to process structured data. 3) Use key frame chunking to process multimedia data.

[0006] Step 1.3: Calculate the data block chunking size according to the system performance metrics of the current cloud environment. The calculation formula for the data block chunking size is: ; In the formula, Y is the chunking size, B is the total bandwidth, N is the number of available nodes, W is the data weight coefficient, is the environment adjustment factor; Step 1.4: The chunking verification unit uses the hash fingerprint technology to verify each data block and establish a chunking identification mapping table. Step 1.5: The chunking optimization feedback loop dynamically adjusts the chunking strategy according to the historical chunking performance data through a reinforcement learning model.

[0007] The reward function of the reinforcement learning model is: ; In the formula, R is the reward function, is the data processing time after chunking, E is the utilization rate of the storage space, F is the chunking failure rate, and are dynamically adjusted weight coefficients.

[0008] Furthermore, the adaptive dynamic chunking engine includes a data stream feature extraction module, a chunking rule generator, a chunking verification unit, and a chunking optimization feedback loop.

[0009] Step 2: According to the data block metadata tags obtained in Step 1 and the multi-dimensional evaluation indicators of the storage nodes collected in real time, construct a node selection model through a comprehensive evaluation algorithm, select storage nodes to form an optimal storage cluster, and based on the load status, the load optimization module dynamically adjusts the resource matching weight coefficient, and writes the verified data block into the optimal storage cluster; The comprehensive evaluation algorithm is the entropy weight TOPSIS algorithm. Calculate the comprehensive scores of each node through the entropy weight TOPSIS algorithm and sort them from large to small. Select the top K nodes with the highest scores to form an optimal storage cluster. The expression of K is: ; In the formula, D is the total data volume, A is the average node capacity, H is the maximum allowable number of nodes, is to take the integer of ;

[0010] The multi-dimensional evaluation indicators include the remaining rate of the current storage space of the node, the historical data access response time, the node failure rate, and the network topology location weight; The load optimization module includes a node load prediction model, a multi-dimensional resource matching degree matrix, and a node combination optimization model; The node load prediction model is an LSTM neural network model. The input is the historical load time series data. Through the weight parameters obtained by training, the load change curve of the predicted node within the future Δt time window is output; The matching dimensions of the multi-dimensional resource matching degree matrix include storage capacity, computing power, and network latency. The comprehensive matching degree calculation formula for each node's resources is: ; In the formula, is the comprehensive matching degree of the i-th node, is the computing power matching degree of the node, is the storage capacity matching degree of the node, is the network latency matching degree of the node, , and β and γ are dynamically adjusted weight coefficients; The node combination optimization model uses the simulated annealing algorithm to obtain the optimal node combination; Assume that the current node combination is , and the next node combination is ; Define the optimization objective function, considering the two important elements of the comprehensive matching degree and the load balance degree. The expression of the optimization objective function is: ; In the formula, is the total objective, is the comprehensive matching degree, is the load balance weight, is the load variance; Iteratively update the node combination state through the criterion acceptance probability. The acceptance probability formula is: ; In the formula, is the difference in the comprehensive matching degree between the current state and the new state, is the temperature parameter that controls the randomness of the search, and k is a constant; The load balance degree is quantified by the load variance of the node combination. The load variance expression is: ; In the formula, is the node load, is the average load, is the total number of samples; Control the convergence of the simulated annealing algorithm through the temperature decay mechanism, and output the optimal task allocation scheme that satisfies the load balance.

[0011] Step 3: Decompose the data processing task into parallel subtask units. According to the optimal storage cluster node distribution determined in Step 2, establish the mapping relationship between the subtask units and the data blocks, construct a bi-objective optimization model including the task execution time and resource utilization rate, solve the bi-objective optimization model, and obtain the optimal solution, that is, the optimal task allocation scheme that satisfies the load balance; The bi-objective optimization model adopts an adaptive genetic algorithm. The bi-objective optimization model includes: Adopt a bi-objective fitness function to evaluate the task allocation scheme. The bi-objective fitness function synthesizes the task execution time and resource utilization rate indicators; The calculation formula of the bi-objective fitness function is: ; In the formula, is the comprehensive task execution time, U is the resource utilization rate, and ω1 and ω2 are dynamic weight coefficients; Dynamically adjust the weight coefficients of time and resource utilization rate according to the system load; Introduce a penalty evaluation mechanism with service level agreement SLA constraints during the iteration process; The fitness evaluation function is: ; In the formula, is the task execution time, is the degree of violation of the SLA constraint, is the penalty coefficient, is the exponential penalty factor.

[0012] For the described adaptive genetic algorithm, The population initialization adopts the greedy matching rule of task resource requirements and node remaining resources to generate a preliminary high-quality solution as the initial state of the population; The chromosome encoding adopts a three-dimensional gene structure, including: The first-dimensional task node mapping defines the mapping relationship between each task and the storage node; The second-dimensional task execution order, where tasks are executed on nodes in the correct order; The third-dimensional resource reservation amount, that is, the resources required for each task execution; The crossover operation has an adaptive probability, and the adaptive probability formula is: ; In the formula, is the crossover probability, g is the current generation number, and G is the total generation number; The mutation operation introduces a directed mutation mechanism, and the directed mutation mechanism guides the selection of mutation directions based on the historical excellent solution library.

[0013] Step 4: Deploy a distributed monitoring agent in the computing node cluster to collect the system resource metrics of each computing node. When the system resource metrics exceed the set threshold, trigger the dynamic rebalancing mechanism, perform resource rebalancing operations, and update the system topology state; The dynamic rebalancing mechanism includes a hierarchical warning module, an elastic scaling module, and a migration decision module; The hierarchical warning module divides the resource utilization rate into a first interval, a second interval, and a third interval in sequence; The first interval triggers the first warning level, and the first warning level triggers the load diversion strategy to reallocate tasks; The second interval triggers the second warning level, and the second warning level triggers the vertical expansion of nodes to increase resources; The third interval triggers the third warning level, and the third warning level triggers cross-region data migration; The threshold range of the first interval is that the resource utilization rate is between 70% and 85%; The threshold range of the second interval is that the resource utilization rate reaches 85% to 95; The threshold range of the third interval is that the resource utilization rate exceeds 95%; The elastic scaling module executes the resource adjustment strategy corresponding to the warning level; The cross-region data migration is equipped with a migration cost calculation model and a sandbox simulation unit; The calculation formula of the migration cost calculation model is: ; In the formula, is the total migration cost, is the time cost of data transmission, is the time cost of task interruption, is the time cost for task reconfiguration, , and are weight coefficients; The sandbox simulation unit builds a virtualized test environment consistent with the production environment, simulates the migration process based on historical load data, monitors and records key indicators such as task migration latency, resource competition conflicts, and abnormal events in real time, and generates a report on optimization suggestions for migration strategies Step 5: Build a multi-level cache system to optimize the data access path, perform data conversion and verification in parallel through the pipeline processing unit, start a multi-node verification mechanism for different data blocks, and push the final result to the user terminal and update the knowledge base.

[0014] The multi-level cache system sets caches at the node level, rack level, and data center level; The pipeline processing unit includes: A pipeline parallel architecture that realizes the overlapping execution of data compression and verification operations; A difference detector that uses a Bloom filter to locate data block anomalies; Adopt an intelligent prefetching strategy, predict and load the data to be processed through the historical access pattern, calculate the access probability of the data block, and the calculation formula for the access probability of the data block is: ; In the formula, is the prefetch probability of the data block , is the access frequency of the data block at time t, and T is the predicted time window.

[0015] Compared with the prior art, the beneficial effects of the present invention include: (1) The present invention optimizes the chunking strategy through a dynamic chunking engine combined with reinforcement learning to achieve adaptive chunking with data feature perception; designs a multi-dimensional resource matching degree model and an improved genetic algorithm to achieve intelligent scheduling of tasks and nodes; introduces a hierarchical early warning system and a migration cost model to dynamically adjust resource allocation and optimize cross-region migration decisions; adopts multi-level caches, a pipeline parallel architecture, and Bloom filter difference detection to improve the result merging efficiency and data consistency. Through multi-dimensional collaborative optimization, the resource utilization rate, load balancing ability, and elastic expansion performance of the system are significantly improved, effectively solving the problems of resource waste, response delay, and insufficient scalability of traditional cloud computing systems under dynamic loads.

[0016] (2) The present invention utilizes a dynamic chunking engine to perceive the characteristics of data streams in real time, and combines a reinforcement learning model to continuously optimize the chunking strategy, significantly reducing storage space fragmentation and redundant data accumulation, and greatly improving the utilization rate of storage space. Based on the adaptive adjustment of the cloud environment performance by the dynamic chunking mechanism, it effectively reduces operation latency and optimizes data read and write efficiency, fundamentally solving the problems of storage resource waste and processing performance bottlenecks caused by traditional fixed chunking strategies.

[0017] (3) The present invention constructs a comprehensive node evaluation model through the entropy weight TOPSIS algorithm, combines the dynamic prediction ability of the LSTM neural network for node load changes, and the global optimization of node combinations by the simulated annealing algorithm, significantly improving task allocation accuracy and resource utilization rate, and realizing the coordinated scheduling of multi-dimensional resources. Based on the multi-dimensional resource matching mechanism with dynamic weight adjustment, it effectively balances the load distribution among nodes, greatly shortens the task response time and reduces resource usage fluctuations, solving the problems of local node overload and resource idleness caused by traditional polling or single-index allocation strategies, and significantly improving the system throughput and stability in high-concurrency scenarios.

[0018] (4) Through the double-objective fitness function in the adaptive genetic algorithm and the dynamic parameter tuning mechanism driven by reinforcement learning, the system can autonomously balance task allocation accuracy and resource usage efficiency, significantly reducing task allocation deviation and stabilizing resource utilization fluctuations, ensuring long-term stable operation in dynamic load scenarios. Combining three-dimensional chromosome coding and directional mutation mechanism, the algorithm quickly converges to the global optimal solution during iteration, while enhancing the system's dynamic adaptability to load changes, thus achieving efficient task scheduling and resource allocation in complex environments, and comprehensively improving the system scalability and operation stability.

[0019] (5) Through the dynamic monitoring and intelligent grading of the resource usage status by the hierarchical early warning magic resistance, and combining the comprehensive evaluation of network transmission, task interruption and reconfiguration overhead by the migration cost calculation model, it realizes the millisecond-level precise triggering of resource adjustment strategies under burst traffic, significantly shortening the system expansion response time and ensuring the high availability of services. At the same time, with the help of the sandbox simulation unit to simulate the whole process of migration and predict risks, it optimizes the migration strategy and avoids potential conflicts, greatly reducing resource waste and stability risks caused by migration while ensuring business continuity, breaking through the decision-making lag and blindness caused by traditional reliance on manual experience to set thresholds or blind migration, and comprehensively improving the elastic response ability and decision-making reliability of the system in complex load scenarios.

[0020] (6) By constructing a multi - level cache system and introducing an intelligent pre - fetching strategy, the system can dynamically predict and load high - frequency data based on historical access patterns, significantly improving data access efficiency and reducing overall latency. The pipeline parallel architecture optimizes the data processing throughput by overlapping the execution of data compression, verification, and merging operations, accelerating the task execution process. At the same time, the difference detection mechanism based on the Bloom filter accurately identifies and repairs the inconsistencies between data blocks while minimizing the consumption of computing resources, effectively avoiding the high cost of traditional full - volume comparison, ensuring high precision and high reliability of data processing results in real - time analysis scenarios, and meeting the dual stringent requirements of the business side for efficiency and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the drawings and embodiments.

[0022] Figure 1 It is a flowchart of a collaborative method for big data distributed storage and parallel processing based on cloud computing in an embodiment of the present invention.

[0023] Figure 2 It is a flowchart of data chunking and metadata annotation in an embodiment of the present invention.

[0024] Figure 3 It is an architecture diagram of a load optimization module in an embodiment of the present invention.

[0025] Figure 4 It is a flowchart of an adaptive genetic algorithm in an embodiment of the present invention.

[0026] Figure 5 It is an architecture diagram of a dynamic re - balancing mechanism in an embodiment of the present invention.

[0027] Figure 6 It is a schematic architecture diagram of a multi - level cache system and a pipeline processing unit in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] As Figure 1 and Figure 2 and Figure 3 and Figure 4 and Figure 5 and Figure 6 As shown, the collaborative method for big data distributed storage and parallel processing based on cloud computing includes the following steps: Step 1: Receive the original data stream, divide the original data stream into data blocks through an adaptive dynamic chunking engine, dynamically calculate the size of the divided data blocks and perform verification, and generate metadata tags; Step 1.1: Extract the data features of the original data stream through a data stream feature extraction module; Step 1.2: The chunking rule generator performs data chunking on the data blocks using an adaptive chunking strategy according to the data characteristics obtained in Step 1.1; The adaptive chunking strategy includes: 1) Process data with entropy value changes exceeding the threshold using a fixed-size chunking method; 2) Record boundary chunking to process structured data; 3) Process multimedia data using key frame chunking.

[0029] Step 1.3: Calculate the data block chunk size according to the system performance metrics of the current cloud environment; The calculation formula for the data block chunk size is: ; In the formula, Y is the chunk size, B is the total bandwidth, N is the number of available nodes, W is the data weight coefficient, is the environment adjustment factor; Step 1.4: The chunk verification unit uses the hash fingerprint technology to verify each data block and establish a chunk identification mapping table; Step 1.5: The chunk optimization feedback loop dynamically adjusts the chunking strategy according to the historical chunking performance data through the reinforcement learning model.

[0030] The reward function of the reinforcement learning model is: ; In the formula, R is the reward function, is the data processing time after chunking, E is the utilization rate of the storage space, F is the chunking failure rate, and are dynamically adjusted weight coefficients.

[0031] The adaptive dynamic chunking engine includes a data flow feature extraction module, a chunking rule generator, a chunk verification unit, and a chunk optimization feedback loop.

[0032] In the adaptive dynamic chunking engine of Step 1, the adaptive chunking strategy based on data feature perception is the core part. First, the raw data stream is analyzed in real time through the data flow feature extraction module to extract the entropy change rate, data type distribution characteristics, and access frequency pattern. The entropy value reflects the uncertainty of the data, and the entropy change rate can indicate the change trend of the data stream over time, which helps to identify hotspots or sudden changes in the data stream. The data type distribution characteristics help to determine the appropriate chunking method by analyzing the structured degree of the data, and the access frequency pattern reveals the access rule of the data in the time dimension, thus affecting the chunking strategy selection.

[0033] Based on these feature analysis results, the chunking rule generator dynamically selects appropriate chunking strategies. For data with high entropy value fluctuations, a fixed-size chunking method can better balance storage efficiency and computational cost, ensuring uniform data chunking and reducing frequent re-chunking. In the case of structured data, a chunking strategy based on record boundaries can maintain data integrity and avoid losing structural information during data storage. For multimedia data, due to its different characteristics, using key-frame chunking is more reasonable, which can ensure that the processing of video or image data not only maintains the temporal order of the data but also avoids redundant processing caused by over-segmentation.

[0034] The chunk verification unit is responsible for verifying the validity of the chunks. It generates a unique identifier for each data chunk using the hash fingerprint technology and establishes a chunk-fingerprint mapping table. This process ensures the consistency and integrity of the data chunks during subsequent processing through hash value comparison, avoiding data loss or errors after chunking. To further improve the adaptability of the chunking strategy, the chunk optimization feedback loop learns and optimizes historical chunking performance data based on a reinforcement learning model. Through the optimization strategy, the reinforcement learning model can dynamically adjust the parameters of the chunking strategy to adapt to changes in different data streams and fluctuations in system load.

[0035] To quantify the optimization process, the reward function of the reinforcement learning model can be evaluated based on multiple dimensions such as the processing time after chunking, storage efficiency, and accuracy of task completion. The reward function is set as: ; where R is the reward function, is the data processing time after chunking, E is the utilization rate of the storage space, F is the chunking failure rate, , and are dynamically adjusted weight coefficients, reflecting the importance of time, space, and error rate in the optimization process. The system can optimize the utilization of storage resources while ensuring the task completion efficiency and reduce the risk of data chunking failure. Through this reward function, the reinforcement learning model continuously adjusts the chunking strategy, thereby adaptively optimizing the chunking process in a complex cloud computing environment to ensure the efficient storage and processing of big data.

[0036] Step 2: According to the data block metadata tags obtained in Step 1 and the multi-dimensional evaluation indicators of the storage nodes collected in real time, construct a node selection model through a comprehensive evaluation algorithm, select storage nodes to form an optimal storage cluster, and dynamically adjust the resource matching weight coefficients based on the load status through the load optimization module, and write the verified data blocks into the optimal storage cluster; The comprehensive evaluation algorithm is the entropy weight TOPSIS algorithm. The comprehensive scores of each node are calculated by the entropy weight TOPSIS algorithm and sorted from large to small. The top K storage nodes with the highest scores are selected to form the optimal storage cluster. The expression of K is: ; where D is the total data volume, A is the average node capacity, H is the maximum allowable number of nodes, is to take the integer.

[0037] The multi-dimensional evaluation indicators include the remaining rate of the current storage space of the storage node, the access response time of historical data, the failure rate of the storage node, and the network topology location weight; The load optimization module includes a node load prediction model, a multi-dimensional resource matching degree matrix, and a node combination optimization model; In the node selection model in step 2, load balancing optimization is a key part to ensure the efficient operation of the cloud environment. First, the node load prediction model analyzes the historical load data of each node through the LSTM neural network to predict the change trend of the load within the future time window. LSTM has the characteristics of long-term and short-term memory and can capture long-term dependencies in time series data. Therefore, it is very suitable for predicting load changes. Specifically, by inputting the historical load data of each node into the LSTM network, the network uses the weight parameters obtained through training to predict the load change curve of the node within the future Δt time window. This prediction process can not only provide a pre-judgment of load changes for storage node selection, but also help the system to perform early scheduling before the load peak.

[0038] Design a multi-dimensional resource matching degree matrix, and match the storage requirements with the resources of each storage node through a multi-objective matching method. The matching dimensions of the multi-dimensional resource matching degree matrix include storage capacity, computing power, and network latency. The storage capacity matching degree measures whether the node can meet the storage requirements of data blocks; the computing power matching degree evaluates the ability of the node to process data tasks; the network latency matching degree ensures that data transmission between nodes can be completed with low latency.

[0039] The formula for the comprehensive matching degree of each storage node resource is: ; where is the comprehensive matching degree of the i-th node, is the computing power matching degree of the node, is the storage capacity matching degree of the node, is the network latency matching degree of the node, , β, and γ are dynamically adjusted weight coefficients, which reflect the relative importance of computing power, storage capacity, and network latency in node selection. The role of this formula is to perform a weighted evaluation of the resources of each node according to the specific requirements of the system, ensuring that the selected nodes can meet multiple requirements of storage, computing, and network latency simultaneously.

[0040] Under the dynamic changes of the system load, the weight coefficients will be adjusted with the fluctuations of the overall load. When the overall system load is below 50%, the matching of computing power is given priority to improve the computing and processing capabilities of the system; when the system load is above 50%, the matching of network latency is emphasized to ensure the high efficiency and low latency of data transmission.

[0041] Establish a node combination optimization model and use the simulated annealing algorithm to find the optimal node combination.

[0042] The simulated annealing algorithm is a probabilistic search algorithm that explores the optimal solution to the problem by simulating the process of a solid substance gradually cooling at high temperature. During this process, the algorithm will continuously adjust the state of the current node combination to gradually optimize the node matching degree and load balancing degree.

[0043] Let the current node combination be , and the next node combination be ; Define the optimization objective function, which combines the two important elements of node matching degree and load balancing degree. The expression of the optimization objective function is: ; In the formula, is the total objective, is the comprehensive matching degree, is the load balancing weight, is the load variance; Update the node combination state through the criterion acceptance probability. The acceptance probability formula is: ; In the formula, is the difference in comprehensive matching degree between the current state and the new state, is the temperature parameter that controls the randomness of the search, and k is a constant; The load balancing degree is quantified by the load variance of the node combination. The load variance expression is: ; In the formula, is the node load, is the average load, is the total number of samples; Control the convergence of the simulated annealing algorithm through the temperature decay mechanism and output the optimal task allocation scheme that meets the load balancing.

[0044] Through the search process of simulated annealing, it is possible to avoid falling into local optimal solutions and ensure that the selected node combination achieves global optimality in terms of load balancing and resource matching.

[0045] Step 3: Decompose the data processing task into parallel subtask units. According to the optimal storage cluster node distribution determined in Step 2, establish the mapping relationship between the subtask units and the data blocks, construct a bi-objective optimization model including task execution time and resource utilization rate, solve the bi-objective optimization model, and obtain the optimal solution, that is, the optimal task allocation scheme that satisfies load balancing. The bi-objective optimization model adopts an adaptive genetic algorithm. The bi-objective optimization model includes: Use a bi-objective fitness function to evaluate the task allocation scheme. The bi-objective fitness function synthesizes the task execution time and resource utilization rate indicators. The calculation formula of the bi-objective fitness function is: ; In the formula, is the comprehensive task execution time, U is the resource utilization rate, ω1 and ω2 are dynamic weight coefficients; dynamically adjust the weight coefficients of time and resource utilization rate according to the system load. The population initialization adopts the greedy matching rule of task resource requirements and node remaining resources, and generates a preliminary high-quality solution as the initial state of the population. The chromosome coding adopts a three-dimensional gene structure, including: The first dimension is task-node mapping, which defines the mapping relationship between each task and the storage node. The second dimension is the task execution order, and the tasks are executed on the nodes in the correct order. The third dimension is the resource reservation amount, that is, the resources required for each task execution. The adaptive probability adjustment of the crossover operation is one of the key optimizations of the algorithm. In the traditional genetic algorithm, the crossover probability is usually fixed, while in this improved algorithm, the crossover probability gradually decreases with the increase of generations. The purpose is to conduct extensive searches in the initial exploration stage and accelerate convergence in the later stage. The adjustment formula of the crossover probability is: ; In the formula, is the crossover probability, g is the current generation number, and G is the total generation number. As the generation number g increases, the crossover probability gradually decreases, making the algorithm gradually focus on the optimization of excellent individuals and avoid excessive exploration. The purpose of this formula is to optimize the search process, balance global search and local optimization by dynamically adjusting the crossover probability, and thus improve the efficiency and accuracy of the algorithm.

[0046] In terms of the mutation operation, the algorithm introduces a directed mutation mechanism. Traditional mutation operations are usually random, while this algorithm guides the mutation direction based on the feature library of historical excellent individuals. Specifically, the mutation operation refers to the resource allocation, task scheduling and other features of the best-performing individual in history to generate mutant solutions that conform to these features. By guiding the mutation, the algorithm can more effectively explore the potential excellent solution space, thereby improving the optimization efficiency and reducing unnecessary invalid mutations.

[0047] The fitness evaluation introduces a penalty term to evaluate the quality of the solution, especially when violating the Service Level Agreement (SLA) constraints. SLA constraints are key requirements in cloud computing, ensuring that tasks are completed within the specified time and resource allocation meets expectations. For solutions that violate SLA constraints, the fitness function imposes an exponential penalty score to ensure that the algorithm always prioritizes meeting these hard requirements during the optimization process. The form of the fitness evaluation function can be expressed as: ; In the formula, is the task execution time, is the degree of violation of SLA constraints, is the penalty coefficient, is the exponential penalty factor.

[0048] Through this formula, individuals that violate SLA will quickly receive greater penalties, forcing the algorithm to prioritize solutions that meet SLA requirements, thereby improving the feasibility and practicality of the solutions.

[0049] Step 4: Deploy distributed monitoring agents in the computing node cluster to collect system resource metrics of each computing node. When the system resource metrics exceed the set threshold, trigger the dynamic rebalancing mechanism, perform resource rebalancing operations and update the system topology status; The dynamic rebalancing mechanism includes a hierarchical warning module, an elastic scaling module and a migration decision module; The hierarchical warning module divides the resource utilization rate into the first interval, the second interval and the third interval in turn; The first interval triggers the first warning level, and the first warning level triggers the load diversion strategy to reallocate tasks; The second interval triggers the second warning level, and the second warning level triggers the vertical expansion of nodes to increase resources; The third interval triggers the third warning level, and the third warning level triggers cross-regional data migration; The first interval indicates that the resource utilization rate is between 70% and 85%, indicating a relatively high system load but still within a controllable range; the second interval indicates that the resource utilization rate reaches 85% to 95%, and the system load is already approaching saturation, and measures need to be initiated for intervention; the third interval represents that the resource utilization rate exceeds 95%, the system load is too heavy, resulting in tasks unable to be executed normally, and urgent countermeasures must be taken.

[0050] Based on different warning levels, the elastic scaling strategy will automatically trigger corresponding countermeasures to achieve timely allocation of resources and load balancing. At the first warning level, the system initiates a load diversion strategy to redistribute tasks and data traffic to nodes or storage units with lighter loads to avoid overloading a single node; at the second warning level, the system initiates vertical scaling to increase the computing and storage resources of a single node to enhance the node's processing capacity; while at the third warning level, the system initiates cross-region data migration to migrate some tasks and data to nodes in other regions to relieve the load in the current region and ensure the stability and high availability of the system.

[0051] The cross-region data migration is equipped with a migration cost calculation model and a sandbox simulation unit; To optimize the decision-making process of cross-region migration, a migration cost calculation model is introduced. This model comprehensively considers data transmission costs, task interruption losses, and reconfiguration overheads to ensure the optimality of migration decisions.

[0052] The calculation formula of the migration cost calculation model is: ; In the formula, is the total migration cost, is the time cost of data transmission, is the time cost of task interruption, is the time cost of task reconfiguration, , and are weight coefficients; the purpose of this formula is to quantify various types of overheads during the migration process to select the most cost-effective migration path and strategy, thereby avoiding unnecessary resource waste and performance degradation.

[0053] The sandbox simulation unit, by constructing a virtualized test environment consistent with the production environment, simulates the migration process based on historical load data, real-time monitors and records key indicators such as task migration latency, resource competition conflicts, and abnormal events, and generates a migration strategy optimization recommendation report Step 5: Construct a multi-level cache system to optimize the data access path, implement data conversion and verification through parallel processing by the pipeline processing unit, initiate a multi-node verification mechanism for different data blocks, output the final result and push it to the user terminal and update the knowledge base.

[0054] Multi - level cache system, by setting caches at the node level, rack level, and data center level; Establish a multi - level cache system. By setting caches at the node level, rack level, and data center level, it effectively reduces data duplicate reads and transmission latency. Node - level caches store frequently accessed data blocks and can quickly provide local data; rack - level caches further improve the cross - node data access efficiency; data center - level caches cache and aggregate global data to ensure that cross - rack data access is not limited by bandwidth. This multi - level caching strategy effectively reduces the waiting time and bandwidth consumption during data access and improves the overall efficiency of result merging.

[0055] The pipeline processing unit includes: A pipeline parallel architecture that enables overlapping execution of data compression and verification operations; A difference detector that uses a Bloom filter to locate data block anomalies; Adopt an intelligent prefetching strategy. By predicting the loading of data to be processed through historical access patterns and calculating the data block access probability, the data block access probability calculation formula is: ; In the formula, is the prefetch probability of data block , is the access frequency of data block at time t, and T is the predicted time window.

[0056] To further improve efficiency, a pipeline parallel architecture is adopted to overlap multiple processes such as result collection, format conversion, data verification, and compression storage. In this architecture, multiple data processing tasks can be executed in parallel at different stages. Before format conversion is completed, verification and compression operations can be started in parallel, avoiding resource idleness during the waiting process. The parallel processing of the pipeline can make full use of multi - core processors and distributed computing resources to accelerate the speed of data processing and merging.

[0057] The implementation of the intelligent prefetching strategy is based on the processing mode of data access, predicting and pre - loading data blocks that may be required by subsequent steps. This strategy analyzes historical data access patterns and combines the real - time calculated load conditions to predict which data blocks will be accessed during the next processing. By pre - loading these data blocks, it can reduce the waiting time during the actual processing of subsequent steps and further reduce the latency of data access.

[0058] In terms of difference detection, a difference detector is deployed, and a Bloom filter is adopted to quickly identify inconsistent data blocks. A Bloom filter is a data structure with high space efficiency and fast query speed. By generating hash values for each data block and storing them in the Bloom filter, when data comparison is performed, it can quickly determine whether a certain data block has differences. By using the Bloom filter during each data merge, it is possible to quickly screen out data blocks that may have problems without fully loading all the data, greatly reducing the computational overhead and time of data comparison. The efficiency of difference detection is controlled by the false positive rate of the Bloom filter, and the false positive rate can be controlled by adjusting the number of hash functions and the size of the bit array to ensure reducing the risk of false positives while ensuring space efficiency.

Claims

1. A collaborative method for big data distributed storage and parallel processing based on cloud computing, characterized in that, It includes the following steps: Step 1: Receive the original data stream, divide the original data stream into data blocks through an adaptive dynamic chunking engine, dynamically calculate the size of the divided data blocks and perform verification, and generate metadata tags; Step 2: According to the data block metadata tags obtained in Step 1 and the multi-dimensional evaluation indicators of the storage nodes collected in real time, construct a node selection model through a comprehensive evaluation algorithm, select storage nodes to form an optimal storage cluster, and dynamically adjust the resource matching weight coefficient based on the load status through a load optimization module, and write the verified data blocks into the optimal storage cluster; Step 3: Decompose the data processing task into parallel subtask units, establish a mapping relationship between the subtask units and the data blocks according to the node distribution of the optimal storage cluster determined in Step 2, construct a two-objective optimization model including task execution time and resource utilization rate, solve the two-objective optimization model, and obtain the optimal solution, that is, the optimal task allocation scheme that satisfies load balancing; Step 4: Deploy a distributed monitoring agent in the computing node cluster, collect the system resource indicators of each computing node, and when the system resource indicators exceed the set threshold, trigger a dynamic rebalancing mechanism, perform resource rebalancing operations and update the system topology status; Step 5: Construct a multi-level cache system to optimize the data access path, parallel process through a pipeline processing unit to achieve data conversion and verification, start a multi-node verification mechanism for different data blocks, output the final result and push it to the user terminal and update the knowledge base.

2. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 1, wherein In Step 1, the adaptive dynamic chunking engine includes a data stream feature extraction module, a chunking rule generator, a chunking verification unit, and a chunking optimization feedback loop. Step 1 includes the following sub-steps: Step 1.1: Extract the data characteristics of the original data stream through the data stream feature extraction module; Step 1.2: The chunking rule generator performs data block chunking using an adaptive chunking strategy according to the data characteristics obtained in Step 1.1; Step 1.3: Calculate the data block chunking size according to the system performance indicators of the current cloud environment; The calculation formula for the data block chunking size is: ； Wherein, Y is the block size, B is the total bandwidth, N is the number of available nodes, W is the data weight coefficient, is the environmental adjustment factor; Step 1.4: The chunking verification unit uses the hash fingerprint technology to verify each data block and establish a chunking identification mapping table; Step 1.5: The chunking optimization feedback loop dynamically adjusts the chunking strategy according to the historical chunking performance data through a reinforcement learning model.

3. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 2, wherein In Step 1.2, the adaptive chunking strategy includes: 1) Process data with entropy value changes exceeding the threshold using a fixed-size chunking method; 2) Record boundary chunking to process structured data; 3) Use key-frame chunking to process multimedia data.

4. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 3, wherein In Step 1.5, the reward function of the reinforcement learning model is: ; where R is the reward function, is the data processing time after chunking, E is the utilization rate of the storage space, F is the chunking failure rate, and are dynamically adjusted weight coefficients.

5. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 4, wherein In Step 2, the comprehensive evaluation algorithm is the entropy weight TOPSIS algorithm. Calculate the comprehensive scores of each node through the entropy weight TOPSIS algorithm and sort them from large to small. Select the top K nodes with the highest scores to form an optimal storage cluster. The expression of K is: ； Where D is the total data volume, A is the average node capacity, and H is the maximum allowable number of nodes, For taking the integer value.

6. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 5, wherein, In Step 2, the multi-dimensional evaluation indicators include the remaining rate of the current storage space of the node, the historical data access response time, the node failure rate, and the network topology location weight; The load optimization module includes a node load prediction model, a multi-dimensional resource matching degree matrix, and a node combination optimization model; The node load prediction model is an LSTM neural network model. The input is historical load time series data, and through the weight parameters obtained by training, it outputs the load change curve of the predicted node within the future Δt time window; The matching dimensions of the multi-dimensional resource matching degree matrix include storage capacity, computing power, and network latency. The comprehensive matching degree calculation formula for each node's resources is: ； In the formula, is the comprehensive matching degree of the i-th node, is the computing power matching degree of the node, is the storage capacity matching degree of the node, is the network latency matching degree of the node, , and β and γ are dynamically adjusted weight coefficients; The node combination optimization model uses the simulated annealing algorithm to obtain the optimal node combination; Assume the current node combination is , and the next node combination is ; Define the optimization objective function, integrating the dual elements of node matching degree and load balancing degree. The expression of the optimization objective function is: ； In the formula, is the total target, is the comprehensive matching degree, is the load balancing weight, is the load variance; Iteratively update the node combination state through the acceptance probability, and the acceptance probability formula is: ; In the formula, is the difference in the comprehensive matching degree between the current state and the new state, is the temperature parameter for controlling the randomness of the search, and k is a constant; The load balancing degree is quantified by the load variance of the node combination. The expression of the load variance is: ； wherein, is the node load, is the average load, is the total number of samples; Control the convergence of the simulated annealing algorithm through the temperature decay mechanism, and output the optimal task allocation scheme that meets the load balance.

7. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 6, wherein In step 3, the dual-objective optimization model uses the adaptive genetic algorithm. The dual-objective optimization model includes: Use the dual-objective fitness function to evaluate the task allocation scheme. The dual-objective fitness function integrates the task execution time and resource utilization rate indicators; The calculation formula of the dual-objective fitness function is: ； In the formula, is the comprehensive task execution time, U is the resource utilization rate, and ω1 and ω2 are dynamic weight coefficients; the weight coefficients of time and resource utilization rate are dynamically adjusted according to the system load; a penalty evaluation mechanism constrained by the service level agreement SLA is introduced during the iteration process; The fitness evaluation function is: ； Wherein, is the task execution time, is the degree of violation of the SLA constraint, is the penalty coefficient, is the exponential penalty factor.

8. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 7, wherein The adaptive genetic algorithm The population initialization adopts the greedy matching rule of task resource requirements and node remaining resources, and generates a preliminary high-quality solution as the initial state of the population; The chromosome coding adopts a three-dimensional gene structure, including: The first dimension is task-node mapping, which defines the mapping relationship between each task and the storage node; The second dimension is the task execution order, and the tasks are executed on the nodes in the correct order; The third dimension is the resource reservation amount, that is, the resources required for each task execution; The crossover operation has an adaptive probability. The adaptive probability formula is: ； Wherein, is the crossover probability, g is the current generation number, and G is the total generation number; The mutation operation introduces a directional mutation mechanism. The directional mutation mechanism guides the selection of the mutation direction based on the historical excellent solution library.

9. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 8, characterized in that, In step 4, the dynamic rebalancing mechanism includes a hierarchical warning module, an elastic scaling module, and a migration decision module; The hierarchical warning module divides the resource utilization rate into the first interval, the second interval, and the third interval in sequence; The first interval triggers the first warning level, and the first warning level triggers the load diversion strategy to reallocate tasks; The second interval triggers the second warning level, and the second warning level triggers the vertical expansion of nodes to increase resources; The third interval triggers the third warning level, and the third warning level triggers cross-regional data migration; The elastic scaling module executes the resource adjustment strategy corresponding to the warning level; The cross-regional data migration is equipped with a migration cost calculation model and a sandbox simulation unit; The calculation formula of the migration cost calculation model is: ； Wherein, is the total migration cost, is the time cost of data transmission, is the time cost of task interruption, is the time cost of task reconfiguration, , and are weight coefficients; The sandbox simulation unit, by constructing a virtualization test environment consistent with the production environment, simulates the migration process based on historical load data, real-time monitors and records key indicators such as task migration delay, resource competition conflicts, and abnormal events, and generates a migration strategy optimization recommendation report.

10. The collaborative method for distributed storage and parallel processing of big data based on cloud computing according to claim 9, wherein In step 5, the multi-layer cache system sets caches at the node level, rack level, and data center level; The pipeline processing unit includes: Pipeline parallel architecture to achieve overlapping execution of data compression and verification operations; Difference detector that uses a Bloom filter to locate data block anomalies; Adopt an intelligent prefetching strategy to predict and load the data to be processed through the historical access pattern, calculate the data block access probability, and the data block access probability calculation formula is: ； Wherein, is the prefetch probability of the data block , is the access frequency of the data block at time t, and T is the predicted time window.

Images