A data analysis method, system, device and medium based on distributed communication

Abstract

The present invention discloses a data analysis method, system, device and medium based on distributed communication, which relates to the field of data processing technology, including: based on a distributed communication network, real-time monitoring of node status and acquisition of node resource status data set, establishing a node performance evaluation model through a machine learning algorithm, dynamically allocating computing resources of data analysis tasks to optimize resource allocation, and designing a task allocation strategy and establishing a distributed data processing framework according to the requirements of the data analysis tasks and node performance; obtaining the final data analysis results through distributed decomposition and execution of computing data analysis tasks, aggregating and collecting distributed computing results and dynamically adjusting and optimizing resource task allocation. The present invention can monitor node status in real time and intelligently adjust resource allocation schemes, and has the advantages of dynamically optimizing task allocation according to the communication status and computing power of the nodes, thereby solving the problems of insufficient dynamic resource management and poor flexibility of task scheduling in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and relates to a data analysis method, system, device and medium based on distributed communication. Background Art

[0002] In the current digital age, data analysis has become a key tool for various industries to improve decision-making quality and optimize operational efficiency. With the diversification of data sources and the wide geographical distribution, traditional centralized data analysis methods face challenges such as data transmission, real-time and privacy protection. Data analysis technology based on distributed communication has emerged. It uses distributed communication networks to achieve on-site data analysis and collaborative aggregation of results. This technology can not only process geographically dispersed data, but also provide global analytical insights while protecting data privacy. With the rapid development of technologies such as 5G and edge computing, data analysis based on distributed communication is increasingly widely used in the fields of Internet of Things, smart cities, multinational enterprises, etc., becoming a key technology in the era of big data.

[0003] In order to achieve efficient data analysis based on distributed communication, researchers have proposed a variety of technical solutions. Existing technologies mainly focus on communication protocol optimization, distributed algorithm design, and data synchronization mechanisms. For example, some methods use lightweight communication protocols to reduce network transmission overhead and improve data exchange efficiency; in terms of distributed algorithm design, common methods include federated learning, distributed optimization algorithms, etc., which attempt to achieve global model training while protecting local data privacy; for data synchronization, some technologies use incremental updates or differential transmission to ensure data consistency between nodes; these methods have achieved data analysis in a distributed environment to a certain extent, and provide basic technical support for collaborative analysis of cross-regional and multi-source data.

[0004] However, the existing data analysis technology based on distributed communication still has some significant shortcomings: first, the dynamic resource management capability is insufficient, and it is difficult to cope with the dynamic changes of nodes in the distributed communication network, resulting in inefficient resource utilization, which can easily cause waste of communication resources or delays in the execution of analysis tasks; second, the task scheduling lacks flexibility, and it is impossible to dynamically adjust the task allocation strategy according to the communication status and computing power of the node, which may cause some nodes to be overloaded with communication pressure while other nodes are idle, affecting the overall analysis efficiency; in addition, the existing technology also has deficiencies in processing heterogeneous data sources and ensuring the consistency of analysis results, and it is difficult to effectively support complex cross-domain data analysis scenarios. These problems seriously restrict the application effect of data analysis technology based on distributed communication in large-scale and complex network environments. At present, no effective solution has been proposed for the problems in the above-mentioned related technologies. Summary of the invention

[0005] In response to the problems existing in the prior art, the present invention provides a data analysis method, system, device and medium based on distributed communication, which have the advantages of being able to monitor node status in real time and intelligently adjust resource allocation plans, while dynamically optimizing task allocation according to the communication status and computing power of the nodes, thereby solving the problems of insufficient dynamic resource management and poor task scheduling flexibility in the prior art, and effectively supporting complex cross-domain data analysis scenarios.

[0006] The present invention is achieved through the following technical solutions:

[0007] A data analysis method based on distributed communication, comprising:

[0008] Based on the distributed communication network, the resource status of each node is monitored in real time, and the node resource status data set is obtained. Based on the node resource status data set, a node performance evaluation model is established using a machine learning algorithm.

[0009] Based on the node performance evaluation model, dynamically allocate computing resources for data analysis tasks to obtain the optimal resource allocation plan; and according to the requirements of data analysis tasks and node performance, obtain task allocation strategies, and establish a distributed data processing framework based on task allocation strategies and optimal resource allocation plans;

[0010] Based on the distributed data processing framework, the data analysis tasks are decomposed and executed in a distributed manner, the distributed computing results are aggregated and collected, and the resource task allocation is dynamically adjusted and optimized to obtain the final data analysis results.

[0011] Preferably, a node performance evaluation model is established, specifically including:

[0012] Based on the distributed communication network, the node built-in monitoring tool is used to monitor and collect the resource status data of each node in real time. The resource status data includes the data of CPU usage, memory occupancy, storage space and bandwidth occupancy, and establish a node resource status data set;

[0013] Based on the adaptive evaluation algorithm, the node resource status data set is cleaned, and the sliding window technology is used to perform time series analysis to obtain the node resource status feature set. Through principal component analysis and feature importance evaluation, the main features of the node resource status feature set are screened to establish the node resource status main feature set.

[0014] Based on the main feature set of node resource status, a node performance evaluation model is established through machine learning algorithm.

[0015] Preferably, based on the main feature set of node resource status, a node performance evaluation model is established through a machine learning algorithm, specifically including:

[0016] Based on the learning framework of the machine learning algorithm, a multi-layer perceptron network structure for node performance evaluation is designed, including input layer, hidden layer and output layer. The optimal number of network layers and the number of neurons in each layer of the multi-layer perceptron network structure are determined through the hyperparameter optimization algorithm to obtain the initial node performance evaluation model.

[0017] The main feature set of node resource status is used to train the initial node performance evaluation model, and the generalization ability of the initial node performance evaluation model is evaluated through the cross-validation method. According to the evaluation results, the initial node performance evaluation model with the best performance is selected as the node performance evaluation model.

[0018] Preferably, a distributed data processing framework is established, specifically including:

[0019] Based on the node performance evaluation model, the node performance and load dynamic evaluation results are output, and resource allocation constraints are constructed. Based on the evolutionary algorithm, computing resources are dynamically allocated to data analysis tasks according to the resource allocation constraints to obtain the optimal resource allocation solution;

[0020] According to the data analysis task requirements and node performance, the initial task allocation strategy is designed to divide the data analysis task into node subtasks suitable for distributed processing, and the best computing node is allocated to each node subtask to obtain the optimized task allocation strategy;

[0021] Based on the optimal resource allocation scheme and the optimized task allocation strategy, a distributed data processing framework is constructed; the distributed data processing framework includes a task scheduler, a data flow management module and a result aggregation mechanism.

[0022] Preferably, the optimized task allocation strategy, the specific acquisition process is:

[0023] Based on the computational complexity and data size of the data analysis task, the data analysis task is divided into multiple node subtasks with appropriate granularity to obtain a task decomposition plan;

[0024] Using the node performance evaluation model, we analyze the processing capacity and current load status of each computing node, establish a node performance scoring system, sort the nodes in descending order according to the scores, and obtain a node performance ranking list;

[0025] According to the task decomposition scheme and the node performance ranking list, the best computing node is matched for each node subtask to obtain the optimized task allocation strategy.

[0026] Preferably, a distributed data processing framework is constructed based on the optimal resource allocation scheme and the optimized task allocation strategy, specifically including:

[0027] Based on the optimal resource allocation scheme, a task scheduler for the distributed data processing framework is built to set resource queues for different types of data analysis tasks;

[0028] Based on the optimized task allocation strategy, a data flow management module of the distributed data processing framework is constructed;

[0029] Based on the optimized task allocation strategy, a result aggregation mechanism for the distributed data processing framework is constructed.

[0030] Preferably, based on a distributed data processing framework, the data analysis tasks are decomposed and executed in a distributed manner, the distributed computing results are collected and dynamically adjusted to optimize resource task allocation, and the final data analysis results are obtained, specifically including:

[0031] Using the task scheduler of the distributed data processing framework, the data analysis task is decomposed into multiple subtasks, and the subtasks are assigned to the corresponding computing nodes according to the optimal resource allocation plan;

[0032] The data flow management module based on the distributed data processing framework coordinates the data transmission and processing between computing nodes and performs the core computing process of data analysis;

[0033] Through the result aggregation mechanism of the distributed data processing framework, the calculation results of each computing node are collected, and the resource task allocation is dynamically adjusted according to the real-time monitoring of resource utilization to obtain the final data analysis results.

[0034] A data analysis system based on distributed communication, used to implement the data analysis method based on distributed communication, comprising:

[0035] Dynamic resource management module, task allocation and scheduling module, and distributed data analysis module;

[0036] The dynamic resource management module is used to monitor the resource status of each node in real time based on the distributed communication network, obtain a node resource status data set, and establish a node performance evaluation model based on the node resource status data set using a machine learning algorithm;

[0037] The task allocation and scheduling module is used to dynamically allocate computing resources for data analysis tasks based on the node performance evaluation model to obtain the optimal resource allocation plan; and to obtain the task allocation strategy according to the requirements of the data analysis tasks and the node performance, and to establish a distributed data processing framework based on the task allocation strategy and the optimal resource allocation plan;

[0038] The distributed data analysis module is used to obtain the final data analysis results by distributing and executing the computing data analysis tasks based on the distributed data processing framework, aggregating and collecting the distributed computing results and dynamically adjusting and optimizing the resource task allocation.

[0039] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data analysis method based on distributed communication when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of a data analysis method based on distributed communication.

[0041] Compared with the prior art, the present invention has the following beneficial technical effects:

[0042] The present invention provides a data analysis method based on distributed communication. By innovatively combining machine learning algorithms with distributed computing, machine learning algorithms are used to construct an adaptive node performance evaluation model and a task allocation strategy based on the needs of data analysis tasks and node performance acquisition, thereby improving dynamic resource management capabilities and flexible task allocation, and realizing efficient, flexible and intelligent large-scale data analysis. By real-time monitoring and analysis of node resource status, the present invention can accurately predict node performance and load, thereby optimizing resource allocation plans. Through task allocation strategies, historical execution data can be learned, and task allocation can be dynamically adjusted to adapt to different types of data analysis needs and changing system environments, providing a new solution for large-scale distributed data analysis, which can effectively cope with complex and changeable data processing scenarios, and bring significant performance improvements and efficiency improvements to big data analysis applications in various industries. At the same time, the present invention realizes efficient dynamic resource management through a machine learning algorithm. In a distributed communication system, the dynamic joining and exit of computing nodes is a common phenomenon, but the prior art lacks an effective strategy to monitor the node status in real time and dynamically adjust resource allocation. The present invention uses a machine learning algorithm to establish a node performance evaluation model, which can accurately predict the performance and load of the node by analyzing the node resource status data collected in real time; this machine learning-based method can capture complex nonlinear relationships and provide more accurate performance evaluation than traditional methods. Through continuous learning and adaptation, the model can adapt to the dynamic changes of the system and realize more accurate resource allocation; in addition, the present invention realizes an efficient data analysis process through a flexible task allocation strategy, which solves the problem that the prior art is difficult to efficiently manage and schedule the migration of intermediate result sets in large-scale data processing, and the task allocation strategy often lacks flexibility and cannot dynamically adjust the task allocation according to the real-time load changes of the node.

[0043] Furthermore, the present invention also uses an evolutionary algorithm to optimize the resource allocation scheme. By simulating the evolutionary process, the resource allocation scheme is continuously improved to adapt to different task requirements and system states. This method of combining machine learning and optimization algorithms greatly improves the efficiency and flexibility of resource management and effectively solves the problems of resource waste and task execution delays.

[0044] Furthermore, the present invention sets resource queues for different types of data analysis tasks through the task scheduler of the distributed data processing framework, and allocates subtasks to corresponding computing nodes according to the optimal resource allocation plan; at the same time, the data flow management module is used to coordinate the data transmission and processing between the computing nodes, and the calculation results of each computing node are collected through the result aggregation mechanism, thereby realizing flexible task scheduling and efficient data analysis process; through the flexible task scheduling mechanism, the present invention can dynamically adjust task allocation according to the real-time load changes of each computing node, avoid node overload and idleness, and thus improve the accuracy of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a flow chart of a data analysis method based on distributed communication according to an embodiment of the present invention;

[0047] Figure 2 The present invention is a block diagram of a data analysis system based on distributed communication according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] According to an embodiment of the present invention, a data analysis method and system based on distributed communication are provided.

[0051] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to one embodiment of the present invention, a data analysis method based on distributed communication is provided, and the data analysis method based on distributed communication includes the following steps:

[0052] S1. Based on a distributed communication network, monitor the resource usage of each node in real time, generate a node resource status data set, and analyze the node resource status data set using an adaptive evaluation algorithm; based on the node resource status data set, use a machine learning algorithm to establish a node performance evaluation model, wherein the node resource status data set includes data on the CPU (central processing unit) usage rate, memory occupancy rate, storage space and bandwidth occupancy rate of each node;

[0053] S2. Use the node performance evaluation model to dynamically allocate computing resources for data analysis tasks, obtain the optimal resource allocation plan, and design task allocation strategies based on data analysis task requirements and node performance, and adaptively adjust task allocation strategies. Establish a distributed data processing framework based on the optimal resource allocation plan and optimized task allocation strategy.

[0054] S3. Based on the distributed data processing framework, the data analysis tasks are decomposed and executed in a distributed manner, the distributed computing results are aggregated and collected, and the resource task allocation is dynamically adjusted and optimized to obtain the final data analysis results.

[0055] In one embodiment, based on a distributed communication network, real-time monitoring of resource usage of each node is performed, a node resource status data set is generated, an adaptive evaluation algorithm is used to analyze the node resource status data set, and based on the node resource status data set, a machine learning algorithm is used to establish a node performance evaluation model, including the following steps:

[0056] S11. According to the distributed communication network, using the monitoring tool built into the node, collect resource status data of each node, wherein the resource status data includes data of CPU usage, memory occupancy, storage space and bandwidth occupancy, and establish a node resource status data set;

[0057] S12. Clean the node resource status data set based on the adaptive evaluation algorithm, and use the sliding window technology to perform time series analysis to obtain the node resource status feature set. Through principal component analysis and feature importance evaluation, screen the main features of the node resource status feature set and establish the node resource status main feature set.

[0058] S13. Based on the main feature set of node resource status, a node performance evaluation model is established using a machine learning algorithm.

[0059] In one embodiment, the node resource status data set is cleaned based on an adaptive evaluation algorithm, and a time series analysis is performed using a sliding window technique to obtain a node resource status feature set. The main features of the node resource status feature set are screened through principal component analysis and feature importance evaluation, and a node resource status main feature set is established, including the following steps:

[0060] S121. Based on data analysis technology, the node resource status data set is structured and processed, and abnormal data points in the node resource status data set are identified and removed through an outlier detection algorithm to obtain a cleaned node resource status data set;

[0061] Specifically, the node resource status data is parsed using a JSON (JavaScript Object Notation) parser to convert unstructured data into a structured data set; JSON (JavaScript Object Notation) is a lightweight data exchange format; through an outlier detection algorithm, the interquartile range (IQR) method is used in this embodiment to identify and remove abnormal data points in the data set, and the cleaned data set will include indicators such as CPU (central processing unit) usage, memory usage, storage space, and bandwidth usage, forming a cleaned node resource status data set. IQR is the abbreviation of interquartile range, also known as the interquartile range, which is used to measure the degree of discreteness of data distribution.

[0062] S122, using sliding window technology, performing time series analysis on the cleaned node resource status data set, calculating the statistical characteristics of each node resource status within a preset period, including the average value, standard deviation and trend index, and performing standardization processing to obtain a node resource status feature set;

[0063] S123. According to the node resource status feature set, the principal component analysis method is used to extract the main features, and based on the feature importance evaluation algorithm, several main features with the greatest feature importance are screened out to establish the node resource status main feature set that can explain the data variability.

[0064] In one embodiment, based on the main feature set of node resource status, a node performance evaluation model is established using a machine learning algorithm, including the following steps:

[0065] S131. Based on the learning framework of the machine learning algorithm, design a multi-layer perceptron network structure for node performance evaluation, including an input layer, a hidden layer, and an output layer, and determine the optimal number of network layers and the number of neurons in each layer of the multi-layer perceptron network structure through a hyperparameter optimization algorithm to obtain an initial node performance evaluation model;

[0066] S132, using the main feature set of the node resource status, training the initial node performance evaluation model, and evaluating the generalization ability of the initial node performance evaluation model through a cross-validation method, and selecting the initial node performance evaluation model with the best performance as the node performance evaluation model according to the evaluation result;

[0067] S133. Based on the node performance evaluation model, the main feature set of the node resource status is input, and the dynamic evaluation results of the node performance and load are output.

[0068] In one embodiment, based on the node performance evaluation model, dynamically allocating computing resources of data analysis tasks to obtain an optimal resource allocation solution; and according to the requirements of the data analysis tasks and the node performance, obtaining a task allocation strategy, and establishing a distributed data processing framework based on the task allocation strategy and the optimal resource allocation solution, including the following steps:

[0069] S21. Based on the node performance evaluation model, output the dynamic evaluation results of node performance and load, establish resource allocation constraints, and dynamically allocate computing resources for data analysis tasks based on the evolutionary algorithm according to the resource allocation constraints to obtain the optimal resource allocation solution;

[0070] S22. Design a task allocation strategy based on data analysis task requirements and node performance, divide the data analysis task into node subtasks suitable for distributed processing, and allocate the best computing node to each node subtask to obtain an optimized task allocation strategy;

[0071] S23. Based on the optimal resource allocation scheme and the optimized task allocation strategy, a distributed data processing framework is constructed, wherein the distributed data processing framework includes task scheduling, data flow management module and result aggregation mechanism.

[0072] In one embodiment, based on the node performance evaluation model, the dynamic evaluation results of node performance and load are output, resource allocation constraints are established, and based on the evolutionary algorithm, computing resources are dynamically allocated to data analysis tasks according to the resource allocation constraints to obtain the optimal resource allocation solution, including the following steps:

[0073] S211, constructing resource allocation constraints, including node available resources and load balancing requirements, as input parameters of the optimization algorithm according to the node performance and load dynamic evaluation results output by the node performance evaluation model;

[0074] Specifically, the performance score (0-100) and load level (0-1) of each node are obtained from the node performance evaluation model.

[0075] Specifically, set resource allocation constraints:

[0076] a. Available resources of nodes: Based on the performance score, nodes with a score greater than 80 are marked as high-performance nodes and can be allocated more resources;

[0077] b. Load balancing requirement: Set the load level threshold to 0.8. Nodes exceeding this value will be considered high-load nodes and resource allocation will be reduced.

[0078] Specifically, a constraint matrix is ​​constructed, where each row represents a node and the columns include: number of available CPU cores, available memory, available storage space, and current load level.

[0079] Specifically, the constraint matrix is ​​used as an input parameter of the evolutionary algorithm for subsequent resource allocation decisions.

[0080] S212. Based on the principle of evolutionary algorithm, the resource requirements, node performance scores and load levels of data analysis tasks are encoded as individuals, and the node optimization population is generated through crossover and mutation operations, and the node optimization population size and evolutionary generations are set;

[0081] Specifically, we design an individual encoding scheme: each individual represents a resource allocation scheme, containing N genes (N is the number of nodes), and each gene consists of three parts:

[0082] a. Resource requirements for data analysis tasks: number of CPU cores, memory size, and storage space;

[0083] b. Node performance score: a standardized score of 0-100;

[0084] c. Current load level of the node: a normalized value between 0 and 1.

[0085] S213. Utilize the node performance evaluation model to dynamically update the fitness of the node optimization population individuals, and select the optimal resource allocation solution from the optimization results through a multi-objective decision-making method.

[0086] Specifically, in each generation of evolution, the node performance evaluation model in step S13 is used to update the fitness of each individual in the population:

[0087] a. For each resource allocation plan, estimate the task completion time;

[0088] b. Calculate resource utilization and load balancing;

[0089] c. Combining the above indicators, we can get the fitness score of each individual.

[0090] In one embodiment, a task allocation strategy is designed according to data analysis task requirements and node performance, the data analysis task is divided into node subtasks suitable for distributed processing, and the best computing node is allocated to each node subtask to obtain an optimized task allocation strategy, including the following steps:

[0091] S221. Based on the computational complexity and data size of the data analysis task, divide the data analysis task into a plurality of node subtasks with appropriate granularity to obtain a task decomposition plan;

[0092] Specifically, the MapReduce model is used to divide the data analysis task into Map and Reduce stages; MapReduce is a programming model;

[0093] Specifically, the number of Map tasks is determined based on the size of the data set and the number of available nodes, and each Map task processes a data block of a fixed size (such as 64MB).

[0094] Specifically, the number of Reduce tasks is set according to the complexity of the Reduce operation, and in this embodiment is set to 0.95 times the number of nodes.

[0095] S222, using the node performance evaluation model, analyzing the processing capacity and current load status of each computing node, establishing a node performance scoring system, sorting the nodes in descending order according to the scores, and obtaining a node performance ranking list;

[0096] Specifically, the CPU usage, memory occupancy, and network throughput of each computing node are obtained from the node performance evaluation model.

[0097] Specifically, calculate the node performance score:

[0098] Score = (1-CPU usage)*0.4+(1-memory usage)*0.3+network throughput*0.3;

[0099] Before calculation, the network throughput is normalized;

[0100] Specifically, the nodes are sorted in descending order according to the scores to generate a node performance sorting list.

[0101] S223. According to the task decomposition scheme and the node performance ranking list, the best computing node is matched for each node subtask to obtain an optimized task allocation strategy.

[0102] Specifically, the node subtask list is traversed, and a computing node is assigned to each task, including selecting an available node with the highest score in the performance sorting list; and the resource occupancy and score of the node are updated.

[0103] Specifically, if all high-performance nodes have been assigned tasks, a polling method is used to assign nodes to the remaining tasks.

[0104] In one embodiment, the distributed data processing framework is constructed based on the optimal resource allocation scheme and the optimized task allocation strategy. The distributed data processing framework includes a task scheduler, a data flow management module and a result aggregation mechanism, and includes the following steps:

[0105] S231. Based on the optimal resource allocation scheme, a task scheduler for a distributed data processing framework is constructed to set resource queues for different types of data analysis tasks;

[0106] Specifically, choose Apache Mesos or Kubernetes as the container orchestration tool to manage computing nodes and task scheduling.

[0107] Apache Mesos is an open source cluster management software, known as the kernel of distributed systems;

[0108] Kubernetes is an open source system for automating the deployment, scaling, and management of containerized applications.

[0109] Specifically, design a task scheduler to implement the following functions:

[0110] a. Monitor resource usage (CPU, memory, disk space) of each node;

[0111] b. Select the appropriate node for the task based on the task requirements (such as CPU intensive or memory intensive);

[0112] c. Use priority queues to manage tasks and schedule high-priority tasks first;

[0113] d. Implement elastic scaling mechanism and dynamically adjust task allocation according to node load;

[0114] S232, constructing a data flow management module of a distributed data processing framework according to the optimal computing nodes allocated by the optimized task allocation strategy;

[0115] Specifically, Apache Kafka is used as a distributed stream processing platform to implement the publishing and subscription of data streams. Apache Kafka is an open source messaging system project.

[0116] Specifically, the data flow management module is designed, including:

[0117] a. Create Kafka topics to store data at different stages;

[0118] b. Write producer code to publish raw data or intermediate results to the corresponding Kafka topic;

[0119] c. Write consumer code to read data from Kafka topics for further processing;

[0120] d. Implement data stream partitioning and replication mechanisms to ensure high availability and fault tolerance of data.

[0121] S233. Based on the optimal computing nodes allocated by the optimized task allocation strategy, a result aggregation mechanism of the distributed data processing framework is constructed.

[0122] Kafka is a high-throughput distributed publish-subscribe messaging system that handles all the action stream data from consumers in a website.

[0123] Specifically, Apache Spark is used as a distributed computing framework to process and analyze data.

[0124] Specifically, the design result aggregation module includes:

[0125] a. Use Spark's Data Frame API to perform data transformation and aggregation operations;

[0126] b. Use Spark broadcast variables or accumulators to aggregate global data;

[0127] c. Use Spark's persistence mechanism (such as RDD persistence) to store intermediate results and improve computing efficiency;

[0128] Apache Spark is a fast, general-purpose, open source big data processing framework. Spark is a fast, general-purpose, and scalable big data analysis engine. Data Frame API is a programming interface for processing and manipulating structured data.

[0129] RDD (Resilient Distributed Datasets) is a resilient distributed dataset that provides a highly restricted shared memory model.

[0130] d. Write the final results to a distributed file system (such as HDFS) or database for subsequent analysis and reporting; HDFS (Hadoop Distributed File System) is a distributed file system that is mainly used to store and manage massive file information and solve the storage problem of large data files (such as TB or even PB level);

[0131] In one embodiment, based on a distributed data processing framework, by distributing and executing computing data analysis tasks, aggregating and collecting distributed computing results, and dynamically adjusting and optimizing resource task allocation, the final data analysis results are obtained, including the following steps:

[0132] S31, using the task scheduler of the distributed data processing framework to decompose the data analysis task into multiple subtasks, and assign the subtasks to corresponding computing nodes according to the optimal resource allocation plan;

[0133] Specifically, the task scheduler constructed in step S2 is used to split the data analysis task into subtasks such as data preprocessing, feature engineering, and model application.

[0134] Specifically, according to the optimal resource allocation scheme in step S2, computing resources are allocated to each subtask, including:

[0135] a. Use a resource queue management system (such as YARN) to create task queues corresponding to different types of data analysis tasks; YARN stands for Yet Another Resource Negotiator, also known as another resource coordinator, which is a general resource management system;

[0136] b. Submit the subtask to the corresponding resource task queue through the submitJob() method of the task scheduler. submitJob() is a command line tool used to submit jobs to the cluster for execution.

[0137] Specifically, the distributed data processing framework API (such as Spark's Spark Context) is used to initialize the computing environment on the assigned nodes. Spark Context is a connection point between a client program and a Spark cluster. API is an application programming interface, which is called Application Programming Interface in English.

[0138] S32, a data flow management module based on a distributed data processing framework, coordinates data transmission and processing between nodes, and performs the core computing process of data analysis;

[0139] Specifically, the data flow management module constructed in step S23 is used to process the data flow between nodes, including:

[0140] a. Use Kafka topics to achieve data transmission and intermediate result sharing between nodes;

[0141] b. Use Spark Streaming to access Kafka data streams to achieve real-time data processing.

[0142] Spark Streaming is a component in the Apache Spark ecosystem for real-time data stream processing;

[0143] Specifically, the data analysis subtasks are executed on the allocated computing nodes, including:

[0144] a. Use Spark SQL for data preprocessing and feature engineering;

[0145] b. Apply machine learning algorithms in Spark MLlib for model training or prediction.

[0146] Spark SQL is a module in Apache Spark that is used to process structured data;

[0147] Specifically, use distributed caching mechanisms (such as Spark's broadcast variables) to share global parameters and models to improve computing efficiency. Broadcast is an optimization mechanism used to reduce data transmission overhead and improve computing efficiency;

[0148] S33. Through the result aggregation mechanism of the distributed data processing framework, the calculation results of each node are collected, and according to the real-time monitoring of resource utilization, the task allocation is dynamically adjusted to optimize the analysis efficiency and obtain the data analysis results of the final communication.

[0149] Specifically, the result aggregation mechanism designed in S23 is used to summarize the calculation results of each node, including:

[0150] a. Use Spark's reduce() or aggregate() operations to aggregate distributed computing results;

[0151] b. Collect the final results to the driver program using Spark's collect() method.

[0152] Reduce() is a simple aggregation operation used to combine a set of data elements into a single result;

[0153] aggregate() is a powerful aggregation operation that allows users to perform customized aggregation calculations on elements in a resilient distributed dataset.

[0154] collect() is an action that collects the elastic distributed dataset distributed in each node on the cluster into the driver program and returns an array or list.

[0155] like Figure 2 As shown, according to another embodiment of the present invention, a data analysis system based on distributed communication is further provided, and the data analysis system based on distributed communication includes:

[0156] Dynamic resource management module 1, used to monitor the resource status of each node in real time based on a distributed communication network, obtain a node resource status data set, and establish a node performance evaluation model based on the node resource status data set using a machine learning algorithm;

[0157] Task allocation and scheduling module 2 is used to dynamically allocate computing resources for data analysis tasks based on the node performance evaluation model to obtain the optimal resource allocation plan; and to obtain task allocation strategies according to the requirements of data analysis tasks and node performance, and to establish a distributed data processing framework based on the task allocation strategies and the optimal resource allocation plan;

[0158] The distributed data analysis module 3 is used to obtain the final data analysis results by distributing and executing the computing data analysis tasks based on the distributed data processing framework, aggregating and collecting the distributed computing results and dynamically adjusting and optimizing the resource task allocation.

[0159] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the data analysis method of distributed communication.

[0160] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here may include both a built-in storage medium in a computer device and an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the data analysis method for distributed communication in the above embodiment.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific implementation modes of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A data analysis method based on distributed communication, characterized in that: include: Based on the distributed communication network, the resource status of each node is monitored in real time, and the node resource status data set is obtained. Based on the node resource status data set, a node performance evaluation model is established using a machine learning algorithm. Based on the node performance evaluation model, dynamically allocate computing resources for data analysis tasks to obtain the optimal resource allocation solution; And according to the needs of data analysis tasks and node performance, obtain the task allocation strategy, and establish a distributed data processing framework based on the task allocation strategy and optimal resource allocation plan; Based on the distributed data processing framework, the data analysis tasks are decomposed and executed in a distributed manner, the distributed computing results are collected and dynamically adjusted to optimize resource task allocation to obtain the final data analysis results. Among them, the node performance evaluation model is established, which specifically includes: Based on the distributed communication network, the node built-in monitoring tool is used to monitor and collect the resource status data of each node in real time. The resource status data includes the data of CPU usage, memory occupancy, storage space and bandwidth occupancy, and establish a node resource status data set; Based on the adaptive evaluation algorithm, the node resource status data set is cleaned, and the sliding window technology is used to perform time series analysis to obtain the node resource status feature set. Through principal component analysis and feature importance evaluation, the main features of the node resource status feature set are screened to establish the node resource status main feature set. Based on the main feature set of the node resource status, a node performance evaluation model is established through a machine learning algorithm, specifically including: based on the learning framework of the machine learning algorithm, a multi-layer perceptron network structure for node performance evaluation is designed, including an input layer, a hidden layer, and an output layer, and the optimal number of network layers and the number of neurons in each layer of the multi-layer perceptron network structure are determined through a hyperparameter optimization algorithm to obtain an initial node performance evaluation model; using the main feature set of the node resource status, the initial node performance evaluation model is trained, and the generalization ability of the initial node performance evaluation model is evaluated through a cross-validation method, and the initial node performance evaluation model with the best performance is selected as the node performance evaluation model according to the evaluation results; Among them, the establishment of a distributed data processing framework includes: Based on the node performance evaluation model, the node performance and load dynamic evaluation results are output, and resource allocation constraints are constructed. Based on the evolutionary algorithm, computing resources are dynamically allocated to data analysis tasks according to the resource allocation constraints to obtain the optimal resource allocation solution; According to the data analysis task requirements and node performance, the initial task allocation strategy is designed to divide the data analysis task into node subtasks suitable for distributed processing, and the best computing node is allocated to each node subtask to obtain the optimized task allocation strategy; Based on the optimal resource allocation scheme and the optimized task allocation strategy, a distributed data processing framework is constructed; the distributed data processing framework includes a task scheduler, a data flow management module and a result aggregation mechanism; Among them, based on the optimal resource allocation plan and optimized task allocation strategy, a distributed data processing framework is built, which specifically includes: Based on the optimal resource allocation scheme, a task scheduler for the distributed data processing framework is built to set resource queues for different types of data analysis tasks; Based on the optimized task allocation strategy, a data flow management module of the distributed data processing framework is constructed; Based on the optimized task allocation strategy, a result aggregation mechanism for the distributed data processing framework is constructed; Among them, based on the distributed data processing framework, by distributing and executing computing data analysis tasks, aggregating and collecting distributed computing results and dynamically adjusting and optimizing resource task allocation, the final data analysis results are obtained, including: Using the task scheduler of the distributed data processing framework, the data analysis task is decomposed into multiple subtasks, and the subtasks are assigned to the corresponding computing nodes according to the optimal resource allocation plan; The data flow management module based on the distributed data processing framework coordinates the data transmission and processing between computing nodes and performs the core computing process of data analysis; Through the result aggregation mechanism of the distributed data processing framework, the calculation results of each computing node are collected, and the resource task allocation is dynamically adjusted according to the real-time monitoring of resource utilization to obtain the final data analysis results.

2. A data analysis method based on distributed communication according to claim 1, characterized in that: The optimized task allocation strategy, the specific acquisition process is: Based on the computational complexity and data size of the data analysis task, the data analysis task is divided into multiple node subtasks with appropriate granularity to obtain a task decomposition plan; Using the node performance evaluation model, we analyze the processing capacity and current load status of each computing node, establish a node performance scoring system, sort the nodes in descending order according to the scores, and obtain a node performance ranking list; According to the task decomposition scheme and the node performance ranking list, the best computing node is matched for each node subtask to obtain the optimized task allocation strategy.

3. A data analysis system based on distributed communication, used to implement a data analysis method based on distributed communication as described in any one of claims 1-2, characterized in that: include: Dynamic resource management module, task allocation and scheduling module, and distributed data analysis module; The dynamic resource management module is used to monitor the resource status of each node in real time based on the distributed communication network, obtain a node resource status data set, and establish a node performance evaluation model based on the node resource status data set using a machine learning algorithm; The task allocation and scheduling module is used to dynamically allocate computing resources for data analysis tasks based on the node performance evaluation model to obtain the optimal resource allocation solution; And according to the needs of data analysis tasks and node performance, obtain the task allocation strategy, and establish a distributed data processing framework based on the task allocation strategy and optimal resource allocation plan; The distributed data analysis module is used to obtain the final data analysis results by distributing and executing the computing data analysis tasks based on the distributed data processing framework, aggregating and collecting the distributed computing results and dynamically adjusting and optimizing the resource task allocation.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a data analysis method based on distributed communication as described in any one of claims 1-2 are implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a data analysis method based on distributed communication as described in any one of claims 1-2 are implemented.

Images