Big data processing method and system based on distributed storage

Abstract

The invention discloses a big data processing method and system based on distributed storage, and relates to the technical field of big data processing, and the method comprises the steps: collecting and preprocessing multi-source data, calculating a feature importance score by using an SHAP algorithm, generating a composite hash key by using a CMCHA algorithm, generating a copy, mapping the copy to a node, and carrying out hybrid optimization through a PSO algorithm and an ACO algorithm. AIS is used for verification, a multi-stage feedback neural controller is used for generalization, a KNN model, a Prophet model and a Transform model are used for prediction, global model parameters are aggregated through FedAvg, VAE is used for compressing anonymized data, and bandwidth is allocated based on the game theory. According to the method, the processing efficiency and the reliability of multi-source data are improved through mixed optimization of the PSO algorithm and the ACO algorithm in combination with AIS verification, and the calculation efficiency and the resource utilization rate are improved through combination of multiple models with the VAE and the game theory.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data processing method and system based on distributed storage. Background Art

[0002] With the rapid development of big data technology, distributed storage systems have become the core infrastructure for processing massive amounts of data. Traditional distributed storage technologies achieve high throughput and fault tolerance through data sharding and replication mechanisms. Machine learning methods such as SHAP, PCA, and random forests have been widely used in feature selection and dimensionality reduction. Technologies such as consistent hashing, ACO, and PSO provide a variety of solutions for data distribution strategies. The Prophet model and Transformer architecture demonstrate superior performance in resource demand forecasting and network latency estimation, while the FedAvg algorithm in federated learning effectively solves the problem of model aggregation in distributed environments. These technological advances are jointly driving the big data processing paradigm towards intelligence and adaptability.

[0003] Existing big data processing methods based on distributed storage still have shortcomings. Although PCA and random forests can achieve dimensionality reduction and feature selection, they lack the ability to dynamically evaluate feature importance and are difficult to adapt to real-time changes in data distribution. Existing methods rely on static or simple heuristic algorithms and lack comprehensive consideration of global optimization and dynamic adaptation. Independently used KNN and Prophet models have local optimality problems in task priority assessment, and traditional bandwidth allocation methods do not consider multi-party competitive game relationships, which restricts the performance of big data processing systems in dynamic heterogeneous environments. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a big data processing method and system based on distributed storage, which solves the problem that although PCA and random forest can achieve dimensionality reduction and feature selection, they lack the ability to dynamically evaluate feature importance and are difficult to adapt to real-time changes in data distribution. Existing methods rely on static or simple heuristic algorithms and lack comprehensive consideration of global optimization and dynamic adaptation. The independently used KNN and Prophet models have local optimality problems in task priority evaluation, and the traditional bandwidth allocation method does not consider the multi-party competitive game relationship, which restricts the performance of big data processing systems in dynamic heterogeneous environments.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a big data processing method based on distributed storage, which includes collecting multi-source data and preprocessing it, using the SHAP algorithm to calculate feature importance scores, performing dimensionality reduction and generating dynamic weights through PCA and random forest, combining EWMA to calculate the time window update frequency, using the CMCHA algorithm to generate a composite hash key, and generating copies to map to nodes, generating a final allocation plan through hybrid optimization of the PSO algorithm and the ACO algorithm, combining OBL to calculate opposing nodes, using AIS for verification, generalizing through a multi-level feedback neural controller, decomposing and generating an initial subtask set through a Spark task parser, using a KNN model and a Prophet model to predict priority scores and resource requirements, and generating predicted priority probabilities through a Softmax function, using a Transformer model to predict path delay and bandwidth, aggregating global model parameters through FedAvg, using VAE to compress anonymized data, allocating bandwidth based on game theory, calculating anomaly scores through Isolation Forest, and setting a recovery mechanism, aggregating the data generated by the analysis, and generating a comprehensive report.

[0007] As a preferred solution of the distributed storage-based big data processing method of the present invention, the CMCHA algorithm is used to generate a composite hash key, and a copy is generated and mapped to the node. The final allocation scheme is generated by hybrid optimization of the PSO algorithm and the ACO algorithm, and the OBL is combined to calculate the opposing nodes, including: Use the SHAP algorithm to generate feature importance scores, concatenate them into feature importance score vectors, and assign dynamic weights through random forest to generate dynamic weight vectors; Use EWMA to generate time windows, calculate the update frequency of each time window by averaging, perform weighted fusion of the update frequency and feature importance score, generate virtual signal strength, and splice the virtual signal strength vector; Use the CMCHA algorithm to generate composite hash keys, source ID hash keys, and timestamp hash keys; Based on the composite hash key, a distributed hash ring is generated using the consistent hashing algorithm. The values ​​in the feature importance score vector, dynamic weight vector, and virtual signal strength vector are sorted in descending order. The top L high values ​​are selected, where L is the number of values ​​in the three vectors. These values ​​are then mapped to high-performance nodes using a direct mapping method. Aggregate the three vectors mapped to the same high-performance node to generate data blocks, convert them into metadata using JSON, and generate a global metadata table using ZooKeeper. Based on the global metadata table, three copies are generated using the CACHA algorithm; The nodes in the distributed storage system are defined as intelligent agents, and the PSO algorithm is used to generate the optimized node candidate set; Generate optimized allocation path based on ACO algorithm; The candidate set of optimized nodes is defined as the initial search space of the ACO algorithm, and a hybrid optimization allocation scheme is generated through hybrid optimization of the ACO algorithm and the PSO algorithm. Extract hybrid optimization nodes in hybrid optimization allocation scheme based on OBL and calculate the opposite nodes of hybrid optimization nodes; Use the fitness function to calculate the fitness of the hybrid optimization node and the opposing node respectively, select the node with the larger fitness between the hybrid optimization node and the opposing node, allocate the replica to this node, update the hybrid optimization allocation plan, and generate the final replica allocation plan; The final replica allocation plan is used to update the global metadata table through ZooKeeper's ZAB protocol to generate an updated global metadata table.

[0008] As a preferred solution of the distributed storage-based big data processing method of the present invention, wherein: the verification using AIS and generalization through a multi-level feedback neural controller include: The final replica allocation scheme is defined as the antigen. The feature importance score and virtual signal strength in the global metadata table are extracted and updated, and defined as the antigen feature. The initial antibody template is generated. The replica status is verified through AIS and a consistency test report is generated. Based on the consistency detection report, the pheromone mechanism of the ACO algorithm is used to select the synchronization path, synchronized through Spak, and converted into key-value data blocks through the distributed computing framework; The key-value data blocks are sharded and assigned to nodes, and the optimal subclusters are generated using the k-means clustering algorithm; Using an asynchronous communication protocol, nodes exchange optimal subclusters every time t to generate diverse subclusters. Define generalization rules and construct a multi-level feedback neural controller. Based on the diversity subclusters, generalization processing is performed through the multi-level feedback neural controller to generate anonymized data.

[0009] As a preferred solution of the distributed storage-based big data processing method of the present invention, wherein: the prediction is performed using KNN, Prophet, and Transformer models, the anonymized data is compressed using VAE, and the bandwidth is allocated based on game theory, including: Based on the anonymized data and updated global metadata table, the Spark task parser is used to decompose the data into subtasks, and the Spark task parser is used to map the data to generate an initial subtask set. Extract and update the feature importance scores and virtual signal strengths in the global metadata table and concatenate them into a feature vector; Based on the feature vector, the priority score is predicted by KNN; Construct a Prophet model. Based on the initial subtask set and data blocks, use the Prophet model to predict resource requirements, adjust the predicted priority scores, and convert them into predicted priority probabilities using the Softmax function. Based on the data block, the computing requirements and bandwidth requirements are obtained through multiplication operations, combined with the feature vectors, and spliced ​​into the node feature vector; Based on the initial set of subtasks, generate task metadata for each subtask, which is defined as a query key; Build a Transformer model to predict path delay and bandwidth based on node feature vectors and predicted priority probabilities; Define nodes as game participants, define the efficiency of participating in training as strategy, and obtain the utility function value through the utility function; Compress anonymized data through VAE; The characteristics of the time series are predicted using the Prophet model, and the difference between the characteristics of the predicted time series and the characteristics of the initial time series is calculated to obtain the transmission prediction error; Decompress the compressed results and merge cross-node data blocks to generate query results; The node score is obtained by weighted summing of node load and bandwidth, and optimized by utility function. Combined with the predicted priority probability, the predicted priority weighted probability is generated by weighted summing; Allocate bandwidth to nodes using a game theory-based equilibrium strategy; Use ANC to compress the query results, combine the transmission prediction error and the compression results, and splice them into compressed data; Spark's Shuffle mechanism is used to allocate bandwidth for compressed data transmission, which is then decoded using VAE. The transmission prediction error is combined with the addition operation to generate the reconstructed query results.

[0010] As a preferred solution of the distributed storage-based big data processing method of the present invention, the calculation of anomaly scores by Isolation Forest and the setting of a recovery mechanism include: Calculate the anomaly score of the reconstructed query results through Isolation Forest; Set the node status abnormality threshold, filter the maximum value greater than the node abnormal status threshold, combine it with the abnormality score, and generate a comprehensive abnormality value through addition operation; Based on the comprehensive outlier value, the global operation index is calculated, and the global operation index is filtered to be less than the global operation index threshold to trigger recovery.

[0011] As a preferred solution of the distributed storage-based big data processing method of the present invention, the data generated by the analysis is aggregated to generate a comprehensive report, including: Anonymized data, reconstructed query results, anomaly markers, and global operating indices are aggregated through Spark to generate comprehensive reports.

[0012] As a preferred solution of the big data processing method based on distributed storage described in the present invention, the step of collecting multi-source data and preprocessing the data includes: Use multi-source devices to collect multi-source data, deduplicate, standardize, and extract features from the multi-source data.

[0013] In a second aspect, the present invention provides a big data processing system based on distributed storage, comprising: The collection and optimization module is used to collect and preprocess multi-source data, calculate feature importance scores using the SHAP algorithm, perform dimensionality reduction through PCA and generate dynamic weights through random forest, calculate the time window update frequency using EWMA, generate composite hash keys using the CMCHA algorithm, and map replicas to nodes. The final allocation solution is generated through a hybrid optimization of the PSO and ACO algorithms, calculates opposing nodes using OBL, verifies using AIS, and generalizes through a multi-level feedback neural controller. The predictive scheduling module generates an initial set of subtasks through Spark task parser decomposition, predicts priority scores and resource requirements using the KNN and Prophet models, generates predicted priority probabilities using the Softmax function, predicts path latency and bandwidth using the Transformer model, aggregates global model parameters using FedAvg, compresses and anonymizes data using VAE, and allocates bandwidth based on game theory. Anomaly recovery module, which is used to calculate anomaly scores through Isolation Forest and set up recovery mechanisms; The report generation module is used to aggregate the data generated by the analysis and generate a comprehensive report.

[0014] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the big data processing method based on distributed storage as described in the first aspect of the present invention is implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the big data processing method based on distributed storage as described in the first aspect of the present invention.

[0016] The beneficial effects of the present invention are as follows: the present invention collects multi-source data and performs preprocessing, uses the SHAP algorithm to calculate feature importance scores, uses the CMCHA algorithm to generate composite hash keys, generates copies and maps them to nodes, performs hybrid optimization through the PSO algorithm and the ACO algorithm, uses AIS for verification, generalizes through a multi-level feedback neural controller, uses the KNN model, the Prophet model and the Transformer model for prediction, aggregates global model parameters through FedAvg, uses VAE to compress and anonymize data, and allocates bandwidth based on game theory; the processing efficiency and reliability of multi-source data are improved, and the computing efficiency and resource utilization are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flowchart of the big data processing method based on distributed storage in Example 1.

[0019] Figure 2 Schematic diagram of the big data processing system based on distributed storage in Example 1.

[0020] Figure 3 This is a schematic diagram of the hash key generation mechanism of the big data processing method based on distributed storage in Example 1.

[0021] Figure 4 Schematic diagram of federated learning parameter aggregation of the big data processing method based on distributed storage in Example 1. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0025] Example 1, reference Figures 1 to 4 , which is the first embodiment of the present invention, provides a big data processing method based on distributed storage, comprising the following steps: S1. Collect and preprocess multi-source data, use the SHAP algorithm to calculate feature importance scores, perform dimensionality reduction through PCA and generate dynamic weights through random forest, combine EWMA to calculate the time window update frequency, use the CMCHA algorithm to generate composite hash keys, and generate replicas to map to nodes, generate the final allocation plan through hybrid optimization of the PSO algorithm and the ACO algorithm, combine OBL to calculate the opposing nodes, use AIS for verification, and generalize through a multi-level feedback neural controller; Specifically, multi-source data is collected and preprocessed, including: Use multi-source devices (including MQTT protocol, Fluentd, SDK, and API) to collect multi-source data (including sensor data, log data, user behavior data, and time series data). Deduplication of sensor data and log data is performed using hash deduplication, and user behavior data and time series data are deduplicated using Bloom filters. Sensor data and time series data are normalized using Z-score normalization, and user behavior data and log data are normalized using min-max normalization. Feature extraction is performed on sensor data and time series data using principal component analysis, feature extraction is performed on log data using bag-of-words models, and time series and frequency domain features are extracted using sliding window statistics and Fourier transform extraction. Use PCA to reduce the dimensionality of features, sort the reduced features in descending order, filter the top K features (based on the Top-K filtering setting), and concatenate them into a reduced dimensionality feature set, where K is the number of features after dimensionality reduction.

[0026] The sensor data and log record data are deduplicated through the hash deduplication method, which realizes fast and low-cost data redundancy filtering, effectively reduces the storage and computing resource occupation of invalid data, improves system processing efficiency and reduces bandwidth and storage pressure. The user behavior data and time series data are deduplicated through the Bloom filter, providing an efficient and low-memory data duplication detection mechanism for large-scale data sets, realizing the rapid screening of a large number of user behavior records and continuous time series, and significantly improving the system's data processing throughput. The sensor data and time series data are standardized through Z-score normalization, realizing the normalization of the data within the scale of mean 0 and variance 1, effectively suppressing the offset effect caused by different sources or dimensional differences. The user behavior data and log record data are normalized through minimum-maximum normalization. Standardization processing is performed to retain the relative distribution relationship of the original data. PCA not only reduces the data dimension and computational complexity, but also retains the key information in the data. The bag-of-words model is used to extract features from log record data, and the quantitative expression of the frequency of keyword occurrence in text data is achieved. Sliding window statistics and Fourier transform are used to extract time series features and frequency domain features, effectively realizing the joint mining of local change trends and periodic features in time series data. Dimensionality reduction processing uses PCA to retain the main information of the data, filter redundant dimensions, and reduce computational complexity. Top-K screening is performed on this basis, which helps to concentrate computing resources to process the most representative feature subsets, significantly reducing the subsequent computing burden, while improving the performance of the model under limited resources, thereby achieving the optimal balance between processing efficiency and feature utilization.

[0027] Furthermore, the CMCHA algorithm is used to generate a composite hash key, and copies are mapped to nodes. The final allocation scheme is generated through a hybrid optimization of the PSO algorithm and the ACO algorithm, combined with the OBL to calculate the opposing nodes, including: Use the SHAP algorithm to calculate the SHAP values ​​of each feature, perform normalization, generate feature importance scores, and concatenate them into a feature importance score vector; Based on the feature importance score vector, dynamic weights are assigned through random forest to generate a dynamic weight vector; Use EWMA to generate time windows, calculate the update frequency of each time window by averaging, perform weighted fusion of the update frequency and feature importance score, perform normalization, generate virtual signal strength, and splice the virtual signal strength vector; Use the CMCHA algorithm to generate a composite hash key (including the feature importance score vector, dynamic weight vector, and virtual signal strength vector), a source ID hash key, and a timestamp hash key (the source ID and timestamp are both obtained from logging data); Based on the composite hash key, a distributed hash ring is generated using the consistent hashing algorithm. The values ​​in the feature importance score vector, dynamic weight vector, and virtual signal strength vector are sorted in descending order. The top L high values ​​are selected (based on the Top-K filtering setting). These are then mapped to high-performance nodes (servers equipped with NVMe SSDs and 100GbE are set as high-performance nodes) using direct mapping. L is the number of values ​​in the feature importance score vector, dynamic weight vector, and virtual signal strength vector. Aggregate the feature importance score vector, dynamic weight vector, and virtual signal strength vector mapped to the same high-performance node to generate a data block, convert it into metadata using JSON, and generate a global metadata table using ZooKeeper; Based on the global metadata table, three replicas are generated using the CACHA algorithm (the source ID hash key is mapped to the neighboring nodes of the high-performance node (obtained based on the proximity of the distributed hash ring) to generate the first replica, the timestamp hash key is mapped to the high-throughput server node (the high-throughput server is set to a high-throughput server node, such as a large-scale NVMe SSD array and a high-density HDD array) to generate the second replica, and the composite hash key is mapped to the high-performance node to generate the third replica); Nodes based on the distributed storage system (representing all storage nodes of the distributed storage system, such as servers in an HDFS cluster) are defined as agents; Based on the agent, the PSO algorithm is used, the agent is defined as a particle, the particle position is initialized, and the fitness function is defined. By simulating the particle motion, the particle speed and position are updated, and the maximum number of iterations is reached and stopped (based on the early stopping method setting). The fitness values ​​are sorted in descending order, and the first A nodes are selected (based on the Top-K selection setting), where A is the number of nodes, to generate the optimized node candidate set. The formula is: , in, For nodes (represented as particles) The fitness value of 、 、 as well as is the weight ( 、 as well as The weights of the four are equal, and the sum of the four weights is 1). 、 as well as Node Initial latency (based on logging data), bandwidth (based on SNMP), and load (based on API). For nodes Virtual signal strength; Based on the ACO algorithm, the pheromone matrix is ​​initialized (representing the allocation preference of replicas to nodes), and a heuristic function is defined. The probability of the ant selecting a path (the replica allocation path between nodes) is calculated using a probability formula. The pheromone is updated based on the pheromone update rule. When the maximum number of iterations is reached (based on the empirical rule), the pheromone is sorted in descending order, and the allocation path with the highest pheromone is selected to generate the optimized allocation path. The formula is: , in, is the heuristic function value, For nodes and nodes The initial bandwidth between For nodes and nodes The initial delay between is a constant; The optimization node candidate set is defined as the initial search space of the ACO algorithm. The optimization allocation path is updated through heuristic functions and pheromone updates (adjusted based on the fitness of PSO). The PSO algorithm is used to adjust the particle speed based on the path cost of the ACO algorithm through the PSO speed update formula. The optimization node candidate set is updated and stopped when the maximum number of iterations is reached (set based on the convergence analysis method). The hybrid optimization allocation scheme is generated. The formula is: , , in, For nodes and nodes The pheromone increment, is a constant, is the total cost of the ant’s path (based on ant path evaluation), To select a path The probability of For nodes To the candidate node The pheromone concentration, For nodes To the candidate node The heuristic function value of is the number of all candidate nodes; Extract the hybrid optimization node in the hybrid optimization allocation scheme based on OBL and calculate the opposite node of the hybrid optimization node. The formula is: , in, Corresponding node, Optimize nodes for hybrids, and is the boundary of the initial search space; Use the fitness function to calculate the fitness of the hybrid optimization node and the opposing node respectively, select the node with the larger fitness between the hybrid optimization node and the opposing node, allocate the replica to this node, update the hybrid optimization allocation plan, and generate the final replica allocation plan; The final replica allocation plan is used to update the global metadata table through ZooKeeper's ZAB protocol to generate an updated global metadata table.

[0028] By introducing the SHAP algorithm, the features of multi-source data are scored for their importance one by one, which not only improves the scientific nature of feature selection, but also effectively enhances the responsiveness of subsequent processing stages to high-value features, thereby achieving higher processing efficiency and accuracy. The nonlinear modeling capability of random forests is introduced, and the weights of features are dynamically adjusted according to the interaction between features, effectively improving the robustness and sensitivity of the overall feature selection. EWMA is used to achieve smooth modeling of update trends in time series, and then weighted fusion is performed with feature scores to effectively support dynamic load balancing. The CMCHA algorithm is used to encode feature scores, dynamic weights, and virtual signal strengths into composite hash keys, and the source ID and timestamp are integrated to generate three logical identifiers, which helps to improve the uniqueness and traceability of data in space and time. The composite hash keys are mapped to high-performance nodes to improve the targeting of data block allocation and computing resource utilization, significantly reducing redundant calculation and storage of data on non-optimal nodes. After completing the feature vector mapping, they are uniformly converted into metadata in JSON format, and the global metadata table is generated through ZooKeeper coordination to achieve distributed Consistency control in the system enhances system scalability and stability. By using source ID, timestamp, and composite hash key to correspond to different replica generation strategies, the spatial independence and scenario adaptability of data redundancy are improved, and the system's fault resistance and disaster recovery efficiency are enhanced. Nodes are modeled as intelligent agents, and the PSO algorithm is used to generate an optimized node candidate set, improving the search efficiency in the early stage of allocation and providing a high-quality initial search space for subsequent path planning. The ACO algorithm is used to generate replica allocation paths and a pheromone dynamic adjustment mechanism is introduced, which can accurately schedule resources, reduce task bottlenecks, and improve data transmission efficiency and load balancing. PSO and ACO are combined to generate a hybrid optimization allocation scheme and perform iterative optimization of the search space, which enhances the algorithm's convergence speed and optimal solution stability and overcomes the problem that a single optimization method is prone to falling into local optimality. Based on the OBL theory, the opposing nodes of the hybrid optimization node are calculated and fitness comparison and screening are performed, making the final solution more robust and accurate, effectively avoiding performance degradation caused by local optimality. Finally, the global metadata table is updated through the ZAB protocol, improving the overall stability and business continuity of the system.

[0029] Furthermore, we use AIS for verification and generalization through a multi-level feedback neural controller, including: The final replica allocation scheme is defined as the antigen. The feature importance score and virtual signal strength in the global metadata table are extracted and updated, defined as the antigen feature. The initial antibody template is generated. The replica status is verified through AIS, and a consistency detection report (including the normal and abnormal status of the replica) is generated. Based on the consistency detection report, the pheromone mechanism of the ACO algorithm is used to select the synchronization path and synchronize through Spak; Extract the synchronized data blocks and convert them into key-value pair data blocks through the distributed computing framework; The key-value data blocks are sharded and assigned to nodes, and the optimal subclusters are generated using the k-means clustering algorithm; Using an asynchronous communication protocol, nodes exchange optimal subclusters every time t to generate diverse subclusters. Define generalization rules (including feature fuzzification, category merging, timestamp precision reduction, and subcluster label replacement) and build a multi-level feedback neural controller, including input layer, perception layer, decision layer, execution layer, and output layer; Train a multi-level feedback neural controller using the D4RL dataset; Based on the diversity subclusters, generalization processing is performed through a multi-level feedback neural controller to generate anonymized data.

[0030] By defining the final replica allocation scheme as antigen, extracting and updating the feature importance score and virtual signal strength in the global metadata table, defining them as antigen features, generating an initial antibody template, and using AIS to verify the replica status and generate a consistency detection report, the antigen-antibody matching mechanism in the immune system is introduced into the replica consistency verification process. With the help of feature importance and signal strength, an accurate status description is constructed, and the adaptive recognition capability of AIS is used to efficiently detect and identify anomalies on replicas, which improves the error recognition accuracy and system fault tolerance in the replica management process, and lays a high-quality reliability foundation for subsequent synchronization and data distribution. Based on the consistency detection report, the pheromone mechanism of the ACO algorithm is used to select the synchronization path, and Spark is used for data synchronization, which optimizes network bandwidth utilization and node load balancing, enhances the scalability and real-time performance of the system in large-scale dynamic data environments, and extracts the synchronized data blocks and converts them into key-value pairs of data blocks. The operation of the entire system is improved in the face of The data access efficiency and fault tolerance under high concurrent read and write requests are improved by sharding the key-value data blocks and using the k-means clustering algorithm to generate the optimal sub-clusters, reducing data redundancy and communication overhead between nodes, and improving the local computing efficiency and overall throughput of the system. Through the asynchronous communication protocol, the optimal sub-clusters are exchanged between nodes at intervals of t to generate diverse sub-clusters, realizing the asynchronous information exchange mechanism between nodes and dynamic reconstruction of sub-clusters, effectively enhancing the diversity of sub-cluster sets and system adaptability, thereby enhancing the system's robustness and fault tolerance to changing environments. By defining generalization rules and constructing a multi-level feedback neural controller, the model's portability and data reusability are enhanced, providing high-quality input for subsequent federated learning and model training. Generalization processing is performed through a multi-level feedback neural controller based on diverse sub-clusters, and finally anonymized data is generated, effectively ensuring the privacy and legitimacy of data during sharing, distribution, training, etc., while retaining sufficient statistical and structural information for subsequent modeling and analysis.

[0031] S2: Generate an initial set of subtasks through Spark task parser decomposition, use KNN and Prophet models to predict priority scores and resource requirements, generate predicted priority probabilities through the Softmax function, use the Transformer model to predict path latency and bandwidth, aggregate global model parameters through FedAvg, compress and anonymize data using VAE, and allocate bandwidth based on game theory. Specifically, we use KNN, Prophet, and Transformer models for prediction, use VAE to compress and anonymize data, and allocate bandwidth based on game theory, including: Based on the anonymized data and updated global metadata table, the Spark task parser is used to decompose the data into subtasks (such as filtering, aggregation, and sorting), and the Spark task parser is used to map the data to generate an initial set of subtasks. Extract and update the feature importance scores and virtual signal strengths in the global metadata table and concatenate them into a feature vector; Based on the feature vector, the priority score is predicted by KNN; Build the Prophet model, including the input layer, core processing layer, and output layer; Use the GÉANT Network dataset to train the Prophet model; Based on the initial subtask set and data block, the resource requirements are predicted by the Prophet model, and the predicted priority score is adjusted and converted into the predicted priority probability through the Softmax function. The formula is: , in, For the task The adjusted prediction priority score value of For the task The predicted priority score of For the task In time The predicted resource requirements, is a constant; Based on the data block, the computing requirements and bandwidth requirements are obtained through multiplication operations; Based on the initial set of subtasks, generate task metadata for each subtask (including data block, source ID, predicted priority probability, task type, computing requirements, and bandwidth requirements); Generate query keys (including data block, source ID, and timestamp) based on task metadata; Combine the computing requirements and bandwidth requirements with the feature vectors to form a node feature vector; Build a Transformer model, including input layer, encoder layer, decoder layer, and output layer; Use the CMU ENET dataset to train the Transformer model; Based on node feature vectors and predicted priority probabilities, the Transformer model is used to predict path delay and bandwidth. Define nodes as game participants, define the efficiency of participating in training as strategy, and obtain the utility function value through the utility function; Based on the utility function value, FedAvg is used for aggregation, and the formula is: , in, is the aggregated global Transformer model parameter, is the subset of nodes participating in the federation aggregation (based on random sampling), For nodes The weight factor of (obtained through the utility function), For nodes The number of samples in the local dataset (indicates the number of data for training the Transformer model), For nodes The local model weights (representing the Transformer model at the node Parameters obtained from training above), is the number of samples in the local dataset of participating node u (obtained based on random selection), is the weight factor of participating node u (obtained through the utility function); Based on the aggregated global Transformer model parameters, the optimal path is selected. The formula is: , in, For the task The optimal path selected, For path The predicted path delay, is the weight parameter (based on statistical analysis), For path The predicted bandwidth, To predict the priority probability; Compress anonymized data through VAE; The characteristics of the time series are predicted using the Prophet model, and the difference between the characteristics of the predicted time series and the characteristics of the initial time series (obtained based on the log record data) is calculated to obtain the transmission prediction error; Based on the predicted priority probability, the compression results are distributed to generate compressed data blocks (including compression results, transmission prediction errors, and distribution results). The formula is: , in, For the task , To round the calculation results, is the basic number of bits (obtained based on the compression result), is a dynamic adjustment coefficient (based on the experience of task scheduling); Decompress the compressed result and merge the cross-node data blocks to generate the query result. The formula is: H, in, is the decompressed data block, For the task The latent variables (obtained based on VAE), is the parameter of VAE, H is the transmission prediction error; The node score is obtained by weighted summing of node load and bandwidth, and optimized by utility function. Combined with the predicted priority probability, the predicted priority weighted probability is generated by weighted summing; Based on the equilibrium strategy of game theory, bandwidth is allocated to nodes. The formula is: , in, To allocate bandwidth, is the total system bandwidth (obtained by multiplying the node and link bandwidths obtained from the GÉANT Network dataset); Use ANC to compress the query results, combine the transmission prediction error and the compression results, and splice them into compressed data; Spark's Shuffle mechanism is used to allocate bandwidth for compressed data transmission, which is then decoded using VAE. The transmission prediction error is combined with the addition operation to generate the reconstructed query results.

[0032] The Spark task parser is used to decompose anonymized data and generate an initial subtask set. The feature importance score and virtual signal strength in the global metadata are extracted and spliced ​​to generate feature vectors. This realizes the automatic parsing and key feature extraction of large-scale distributed tasks, significantly improves the system's adaptability to complex data scenarios and the parallelism of task processing, and effectively improves the efficiency and accuracy of the task initialization phase. The KNN model is used to predict the priority score of the feature vector, and the resource demand is predicted in combination with the constructed and trained Prophet model. The result is converted into the predicted priority probability through the Softmax function, which realizes the joint modeling and dynamic adjustment of task execution priority and resource demand. Better task scheduling strategies and improved resource utilization. Based on the subtask set, task metadata including source, priority probability, computing and bandwidth requirements are generated, and query keys are generated for task tracking and scheduling, which realizes the refined management and tracking of tasks throughout the entire life cycle, provides structured support for subsequent model prediction and resource allocation, and improves the controllability and transparency of the system for complex task environments. The Transformer model is constructed and trained, and the node feature vector and priority probability are input to predict path delay and bandwidth to complete the dynamic modeling of the network transmission environment. The Transformer model has outstanding advantages in processing time dependence and high-dimensional input features. Combined with the federated learning strategy training After training, it can fully integrate the local knowledge of distributed nodes, and ultimately achieve more accurate path selection, significantly reducing task scheduling delays and network congestion risks. By introducing the concept of game participants and the utility function mechanism with node training efficiency as the strategy, the model is optimized globally while maintaining data privacy, improving the generalization ability of the model and achieving more fair and reasonable optimization results in resource scheduling scenarios. Based on the aggregated Transformer model output, a comprehensive scheduling function is constructed using predicted delay, bandwidth, and priority probability to select the optimal path for task execution, thereby improving the robustness of path selection and maintaining the stability and efficiency of task execution in a changing network environment. Based on VAE data Compression and anonymization processing not only compresses the data transmission scale and protects user privacy, but also maintains data availability through error control, ultimately realizing an efficient and secure data transmission mechanism. Data reconstruction and decompression processing based on prediction errors enhances the system's integrity protection and recovery capabilities in the data compression-transmission-reconstruction process, ultimately improving the query response quality. The node scheduling scoring and game theory bandwidth allocation mechanism avoid node resource competition and local optimal problems, and improve the overall system throughput and fairness. The compression transmission mechanism based on ANC and SparkShuffle achieves the triple guarantee of high throughput, high compression ratio and high restoration, improving the system's processing efficiency and accuracy in large-scale data transmission.

[0033] S3. Calculate the anomaly score through Isolation Forest and set the recovery mechanism; Specifically, we use Isolation Forest to calculate the anomaly score and set up a recovery mechanism, including: The reconstructed query results are normalized and multi-scale features are extracted using DWT. The anomaly score is calculated using IsolationForest. Results with scores greater than the anomaly score threshold (set based on log data analysis) are filtered out and marked as anomalies. Normalize the node load and bandwidth, calculate the maximum value of the two after normalization, set the node status abnormality threshold, filter out the maximum value that is greater than the node abnormal status threshold (based on the analysis of log data), combine it with the abnormality score, and generate a comprehensive abnormality value through addition operation; Based on the comprehensive outlier value, the global operation index is calculated as follows: , in, is the global operation index, is the number of nodes, For nodes The combined outlier value of For nodes The weight of (based on the feature importance score); The global operation index is less than the global operation index threshold (based on actual operation analysis settings), triggering recovery; Filter and update the replica corresponding to the maximum importance score in the global metadata table, filter the target node whose node load is greater than the load threshold and the node bandwidth is less than the bandwidth threshold (set based on actual operation experience), copy the replica corresponding to the maximum value to the target node, and recover the abnormal data block.

[0034] Through signal processing and machine learning methods, it achieves accurate capture of minor anomalies and complex trends in query reconstruction results, enhances the system's ability to perceive potential data anomalies, improves detection accuracy, reduces missed reporting rate, and adapts to high-dimensional heterogeneous data environments. Through comprehensive attribution analysis of multi-source features, the system's judgment of abnormal states is more three-dimensional and comprehensive, improving the accuracy and robustness of fault warnings, avoiding misjudgments and missed judgments. It is particularly suitable for distributed environments with complex task interweaving and dynamic load changes. Based on comprehensive anomaly values, the global operation index is calculated, realizing the mapping transformation from local anomalies to global risks, providing basic indicators for macro-level operation and maintenance strategies, effectively supporting the precision control of system-level resource reconstruction and recovery strategies, improving system availability and operational stability, shortening fault response time, reducing manual intervention costs, improving data recovery efficiency and system fault tolerance, minimizing the impact of business interruptions, and enhancing the overall system recovery resilience.

[0035] S4, aggregate the data generated by the analysis and generate a comprehensive report; Specifically, the data generated by the analysis is aggregated to generate a comprehensive report, including: Anonymized data, reconstructed query results, anomaly markers, and global operating indices are aggregated through Spark to generate comprehensive reports.

[0036] The Spark platform is used to aggregate and process encrypted multi-dimensional data, achieving efficient scheduling of computing resources and rapid processing of large-scale data, and ultimately generating a highly integrated comprehensive analysis report. This report not only has the ability to monitor system status, track user behavior, and identify anomalies, but can also be used to support decision-making scenarios such as strategy optimization, resource allocation, and risk warning, achieving the beneficial effect of improving data utilization, strengthening system management, and ensuring an organic balance between privacy protection.

[0037] This embodiment also provides a big data processing system based on distributed storage, including: The collection and optimization module is used to collect and preprocess multi-source data, calculate feature importance scores using the SHAP algorithm, perform dimensionality reduction through PCA and generate dynamic weights through random forest, calculate the time window update frequency using EWMA, generate composite hash keys using the CMCHA algorithm, and map replicas to nodes. The final allocation solution is generated through a hybrid optimization of the PSO and ACO algorithms, calculates opposing nodes using OBL, verifies using AIS, and generalizes through a multi-level feedback neural controller. The predictive scheduling module generates an initial set of subtasks through Spark task parser decomposition, predicts priority scores and resource requirements using the KNN and Prophet models, generates predicted priority probabilities using the Softmax function, predicts path latency and bandwidth using the Transformer model, aggregates global model parameters using FedAvg, compresses and anonymizes data using VAE, and allocates bandwidth based on game theory. Anomaly recovery module, which is used to calculate anomaly scores through Isolation Forest and set up recovery mechanisms; The report generation module is used to aggregate the data generated by the analysis and generate a comprehensive report.

[0038] This embodiment also provides a computer device suitable for the case of a big data processing method based on distributed storage, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions, thereby implementing the big data processing method based on distributed storage proposed in the above embodiment.

[0039] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0040] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the big data processing method based on distributed storage proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0041] In summary, the present invention collects multi-source data and preprocesses it, uses the SHAP algorithm to calculate feature importance scores, uses the CMCHA algorithm to generate composite hash keys, generates copies and maps them to nodes, performs hybrid optimization through the PSO algorithm and the ACO algorithm, uses AIS for verification, generalizes through a multi-level feedback neural controller, uses the KNN model, the Prophet model, and the Transformer model for prediction, aggregates global model parameters through FedAvg, uses VAE to compress and anonymize data, and allocates bandwidth based on game theory; the processing efficiency and reliability of multi-source data are improved, and the computing efficiency and resource utilization are improved.

[0042] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A big data processing method based on distributed storage, characterized by: include, Collect and preprocess multi-source data, use the SHAP algorithm to calculate feature importance scores, perform dimensionality reduction through PCA and generate dynamic weights through random forest, combine EWMA to calculate the time window update frequency, use the CMCHA algorithm to generate composite hash keys, and generate replicas to map to nodes. The final allocation scheme is generated through a hybrid optimization of the PSO algorithm and the ACO algorithm, and the opposing nodes are calculated by combining OBL. The AIS is used for verification and generalization is achieved through a multi-level feedback neural controller. The Spark task parser is used to decompose and generate the initial subtask set. The KNN and Prophet models are used to predict priority scores and resource requirements. The Softmax function is used to generate predicted priority probabilities. The Transformer model is used to predict path latency and bandwidth. Global model parameters are aggregated using FedAvg. VAE is used to compress and anonymize data. Bandwidth is allocated based on game theory. Calculate anomaly scores through Isolation Forest and set up recovery mechanisms; Aggregate the data generated by the analysis to generate comprehensive reports.

2. The method for processing big data based on distributed storage according to claim 1, wherein: The CMCHA algorithm is used to generate a composite hash key, and copies are mapped to nodes. The final allocation scheme is generated by hybrid optimization of the PSO algorithm and the ACO algorithm, and the OBL is combined to calculate the opposing nodes, including: Use the SHAP algorithm to generate feature importance scores, concatenate them into feature importance score vectors, and assign dynamic weights through random forest to generate dynamic weight vectors; Use EWMA to generate time windows, calculate the update frequency of each time window by averaging, perform weighted fusion of the update frequency and feature importance score, generate virtual signal strength, and splice the virtual signal strength vector; Use the CMCHA algorithm to generate composite hash keys, source ID hash keys, and timestamp hash keys; Based on the composite hash key, a distributed hash ring is generated using the consistent hashing algorithm. The values ​​in the feature importance score vector, dynamic weight vector, and virtual signal strength vector are sorted in descending order. The top L high values ​​are selected, where L is the number of values ​​in the three vectors. These values ​​are then mapped to high-performance nodes using a direct mapping method. Aggregate the three vectors mapped to the same high-performance node to generate data blocks, convert them into metadata using JSON, and generate a global metadata table using ZooKeeper. Based on the global metadata table, three copies are generated using the CACHA algorithm; The nodes in the distributed storage system are defined as intelligent agents, and the PSO algorithm is used to generate the optimized node candidate set; Generate optimized allocation path based on ACO algorithm; The candidate set of optimized nodes is defined as the initial search space of the ACO algorithm, and a hybrid optimization allocation scheme is generated through hybrid optimization of the ACO algorithm and the PSO algorithm. Extract hybrid optimization nodes in hybrid optimization allocation scheme based on OBL and calculate the opposite nodes of hybrid optimization nodes; Use the fitness function to calculate the fitness of the hybrid optimization node and the opposing node respectively, select the node with the larger fitness between the hybrid optimization node and the opposing node, allocate the replica to this node, update the hybrid optimization allocation plan, and generate the final replica allocation plan; The final replica allocation plan is used to update the global metadata table through ZooKeeper's ZAB protocol to generate an updated global metadata table.

3. The method for processing big data based on distributed storage according to claim 2, wherein: The AIS is used for verification and generalization via a multi-level feedback neural controller, including: The final replica allocation scheme is defined as the antigen. The feature importance score and virtual signal strength in the global metadata table are extracted and updated, and defined as the antigen feature. The initial antibody template is generated. The replica status is verified through AIS and a consistency test report is generated. Based on the consistency detection report, the pheromone mechanism of the ACO algorithm is used to select the synchronization path, synchronized through Spak, and converted into key-value data blocks through the distributed computing framework; The key-value data blocks are sharded and assigned to nodes, and the optimal subclusters are generated using the k-means clustering algorithm; Using an asynchronous communication protocol, nodes exchange optimal subclusters every time t to generate diverse subclusters. Define generalization rules and construct a multi-level feedback neural controller. Based on the diversity subclusters, generalization processing is performed through the multi-level feedback neural controller to generate anonymized data.

4. The method for processing big data based on distributed storage according to claim 3, wherein: The method uses KNN, Prophet, and Transformer models for prediction, uses VAE to compress and anonymize data, and allocates bandwidth based on game theory, including: Based on the anonymized data and updated global metadata table, the Spark task parser is used to decompose the data into subtasks, and the Spark task parser is used to map the data to generate an initial subtask set. Extract and update the feature importance scores and virtual signal strengths in the global metadata table and concatenate them into a feature vector; Based on the feature vector, the priority score is predicted by KNN; Construct a Prophet model. Based on the initial subtask set and data blocks, use the Prophet model to predict resource requirements, adjust the predicted priority scores, and convert them into predicted priority probabilities using the Softmax function. Based on the data block, the computing requirements and bandwidth requirements are obtained through multiplication operations, combined with the feature vectors, and spliced ​​into the node feature vector; Based on the initial set of subtasks, generate task metadata for each subtask, which is defined as a query key; Build a Transformer model to predict path delay and bandwidth based on node feature vectors and predicted priority probabilities; Define nodes as game participants, define the efficiency of participating in training as strategy, and obtain the utility function value through the utility function; Compress anonymized data through VAE; The characteristics of the time series are predicted using the Prophet model, and the difference between the characteristics of the predicted time series and the characteristics of the initial time series is calculated to obtain the transmission prediction error; Decompress the compressed results and merge cross-node data blocks to generate query results; The node score is obtained by weighted summing of node load and bandwidth, and optimized by utility function. Combined with the predicted priority probability, the predicted priority weighted probability is generated by weighted summing; Allocate bandwidth to nodes using a game theory-based equilibrium strategy; Use ANC to compress the query results, combine the transmission prediction error and the compression results, and splice them into compressed data; Spark's Shuffle mechanism is used to allocate bandwidth for compressed data transmission, which is then decoded using VAE. The transmission prediction error is combined with the addition operation to generate the reconstructed query results.

5. The method for processing big data based on distributed storage according to claim 4, wherein: The anomaly score is calculated through Isolation Forest, and a recovery mechanism is set, including: Calculate the anomaly score of the reconstructed query results through Isolation Forest; Set the node status abnormality threshold, filter the maximum value greater than the node abnormal status threshold, combine it with the abnormality score, and generate a comprehensive abnormality value through addition operation; Based on the comprehensive outlier value, the global operation index is calculated, and the global operation index is filtered to be less than the global operation index threshold to trigger recovery.

6. The method for processing big data based on distributed storage according to claim 5, wherein: The data generated by the analysis is aggregated to generate a comprehensive report, including: Anonymized data, reconstructed query results, anomaly markers, and global operating indices are aggregated through Spark to generate comprehensive reports.

7. The method for processing big data based on distributed storage according to claim 1, wherein: The multi-source data collection and pre-processing includes: Use multi-source devices to collect multi-source data, deduplicate, standardize, and extract features from the multi-source data.

8. A big data processing system based on distributed storage, based on the big data processing method based on distributed storage according to any one of claims 1 to 7, characterized in that: include, The collection and optimization module is used to collect and preprocess multi-source data, calculate feature importance scores using the SHAP algorithm, perform dimensionality reduction through PCA and generate dynamic weights through random forest, calculate the time window update frequency using EWMA, generate composite hash keys using the CMCHA algorithm, and map replicas to nodes. The final allocation solution is generated through a hybrid optimization of the PSO and ACO algorithms, calculates opposing nodes using OBL, verifies using AIS, and generalizes through a multi-level feedback neural controller. The predictive scheduling module generates an initial set of subtasks through Spark task parser decomposition, predicts priority scores and resource requirements using the KNN and Prophet models, generates predicted priority probabilities using the Softmax function, predicts path latency and bandwidth using the Transformer model, aggregates global model parameters using FedAvg, compresses and anonymizes data using VAE, and allocates bandwidth based on game theory. Anomaly recovery module, which is used to calculate anomaly scores through Isolation Forest and set up recovery mechanisms; The report generation module is used to aggregate the data generated by the analysis and generate a comprehensive report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the big data processing method based on distributed storage according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the big data processing method based on distributed storage according to any one of claims 1 to 7 are implemented.

Images