A spark platform dynamic resource allocation method for traffic analysis
A dynamic resource and traffic analysis technology, applied in the Internet field, can solve problems such as process failure, resource waste, and freezes, and achieve the effect of reducing the use of memory and processing threads, ensuring reasonable allocation, and ensuring normal operation
Active Publication Date: 2022-05-10
JIANGSU FUTURE NETWORKS INNOVATION
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In addition, the instantaneous surge of traffic data and the huge difference in traffic content caused by a large number of users have brought huge hidden dangers and challenges to the stable operation of traffic processing tasks; but on the other hand, according to different scenarios, different business traffic After pre-collection analysis and research, we found that the changes in traffic data are not completely untraceable. For example, after dividing by time dimensions such as day and night, working days and rest days, the data volume and type complexity of network traffic fluctuate It can always be maintained within a certain range; as mentioned above in the present invention, the scheduler of Spark itself will not dynamically adjust resources for Spark tasks that have allocated resources, which will lead to a larger setting for ensuring the normal processing of Spark tasks. When the number of memory and cores is large, if the amount of data actually processed is small, it may cause other processes on the server to fail to run normally or freeze, resulting in waste of resources
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0035] The specific implementation is as follows:
[0036] Initialize resource configuration process module deployment:
[0037]In the case of successful deployment of the existing Spark platform, the first module of the present invention is deployed. First, you need to configure the server environment: Centos7.5, JDK environment 1.8, Python3, Crontab timing script, Hive database. Crontab timing script is mainly used to trigger monitoring and collect data. The collected historical data is processed by Python tasks and persisted to HDFS. The data analysis module will retrieve the processed data from the HDFS library, and use the collaborative filtering algorithm to find the most similar record in the historical records to the latest configuration, and obtain a more appropriate resource configuration.
[0038] Adaptive Resource Scheduler module deployment:
[0039] When the Spark application starts normally, the Crontab scheduled task will monitor the traffic size, the compl...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
A Spark platform dynamic resource allocation method for traffic analysis, characterized in that it includes parts, wherein the first part: through the Spark resource scheduler, according to the basic performance indicators such as CPU and memory in the server, combined with the complexity of the Spark task process , recommend a more appropriate combination of memory and core configuration; the second part: implement a resource scheduler, by analyzing the CPU, memory, and load network traffic data characteristics of the application program, combined with the ARSA algorithm for resource automatic Adapt to distribution. The automatic scheduling method of Spark resources for traffic analysis can not only adapt the appropriate initial memory and core configuration by analyzing the performance indicators of cluster servers, but also make adaptive adjustments according to the complexity and size of actual traffic, fully Use cluster resources to ensure the stable operation of traffic processing tasks.
Description
technical field [0001] The invention relates to the technical field of the Internet, in particular to a dynamic resource allocation method on a Spark platform aimed at traffic analysis. Background technique [0002] With the rapid development of Internet technology and the rapid expansion of information data, how to deal with massive Internet data has become a technical problem. As a prerequisite for various network operation and security management such as bandwidth management, traffic perspective, attack traceability, virus defense, and intrusion detection, traffic analysis has become the focus of the industry because of its large-scale, diverse, and unstable data. . With the emergence and development of distributed computing, the use of multiple server nodes for parallel processing computing has opened a new door for the processing of massive data, such as distributed processing frameworks such as Hadoop, Spark, and Flink have emerged as the times require. [0003] Apac...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50G06F16/182
CPCG06F9/4881G06F9/5016G06F9/5027G06F9/5061G06F16/182
Inventor 张广兴何旭梁帅
Owner JIANGSU FUTURE NETWORKS INNOVATION