Big data application optimization method and device and storage medium
By collecting real-time load metrics and dynamically allocating resources through reinforcement learning models, combined with hierarchical caching and preloading strategies, the resource configuration of big data applications is optimized, solving the problems of low resource utilization and high response latency, and improving system performance.
Patent Information
- Application Number
- CN202511100858.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-07
- Publication Date
- 2025-11-18
AI Technical Summary
Existing big data application optimization solutions typically employ static resource configuration or single-dimensional performance optimization strategies, resulting in low resource utilization and high response latency.
It adopts real-time load metric collection, a dynamic resource allocation strategy based on reinforcement learning models, and combines a hierarchical caching mechanism and a preloading strategy to optimize the storage architecture and file system, including a load monitoring unit, a scheduling decision unit, and a cache management unit, supporting dynamic scheduling and intelligent cache collaboration.
It significantly improves resource utilization, reduces disk I/O operation latency, and enhances overall system performance, making it particularly suitable for time-sensitive fields such as the Internet of Things and fintech.
Smart Images
Figure CN120973531A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data technology, and in particular to a method, apparatus and storage medium for optimizing big data applications. Background Technology
[0002] As society and the economy continue to develop, the amount of information and data is also growing rapidly. Simultaneously, the storage, integration, and retrieval of this information and data place increasingly higher demands on computer network technology. Therefore, artificial intelligence (AI) has become the most effective method to help people process information and data. In the era of big data, the application of AI technology in computer networks, as a highly efficient and accurate network technology, can leverage its intelligent advantages to rationally classify and process massive amounts of data within computer systems.
[0003] Big data processing involves extracting value from massive amounts of complex data. The most valuable aspect lies in predictive analytics, which uses data mining techniques such as data visualization, statistical pattern recognition, and data description to help data scientists better understand data and make predictive decisions based on the results. Alternatively, it can be used in AI training, where collecting effective big data allows for AI training to obtain predictive decision models for specific tags (such as user interest analysis, online behavior analysis, and sentiment semantic analysis). Therefore, the accuracy of big data collection plays a crucial role in the accuracy of subsequent data mining.
[0004] However, existing big data application optimization solutions typically employ static resource configuration or single-dimensional performance optimization strategies, resulting in low resource utilization and high response latency. Summary of the Invention
[0005] 1. Technical problems to be solved
[0006] The purpose of this invention is to address the problem that existing big data application optimization schemes typically employ static resource configuration or single-dimensional performance optimization strategies, resulting in low resource utilization and high response latency. Therefore, this invention proposes a big data application optimization method, apparatus, and storage medium.
[0007] 2. Technical Solution
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A method for optimizing big data applications includes the following steps:
[0010] S1: Real-time collection of system load metrics, including CPU load, memory load, disk I / O load, and network load. The CPU load includes average load and CPU utilization; the memory load includes physical memory and swap space; the disk I / O load includes read / write throughput and wait queue; and the network load includes bandwidth, number of connections, and packet loss rate.
[0011] S2: Dynamically adjust the resource allocation strategy for computing nodes based on the reinforcement learning model, specifically including the following steps: S2.1: Data acquisition and preprocessing; S2.2: Algorithm selection and training; S2.3: Strategy deployment and online learning;
[0012] S3: Employs a tiered caching mechanism to preload frequently accessed data and reduce disk I / O latency. Specifically, this includes the following steps: S3.1: Cache optimization; S3.2: Storage architecture optimization; S3.3: File system optimization.
[0013] Preferably, in S1, the average load is obtained by using the uptime or top command to get the process queue length within 1 / 5 / 15 minutes, including running processes and processes waiting for I / O; CPU utilization is obtained by using the mpstat or top command to distinguish between user mode (%us), kernel mode (%sy), idle mode (%id), and other sub-indicators.
[0014] The physical memory and swap space: monitor the total amount, used amount, cache and swap space usage using the free command; memory pressure: monitor the buff / cache and available fields to determine if an OOM (Out of Memory) error is triggered;
[0015] Read / write throughput: Use iostat to view disk read / write speed (MB / s) and IOPS; Wait queue: Monitor %util (device utilization) and await (average wait time) to identify bottlenecks;
[0016] Bandwidth and connection count: Real-time traffic monitoring using iftop or nload, and TCP connection status viewed using the ss command; Packet loss rate: Network stability analyzed using ping and tcpdump.
[0017] Preferably, the data acquisition and preprocessing in S2.1 includes: a toolchain: Prometheus for acquiring system metrics, Kafka for real-time data transmission, and Spark / Flink for window aggregation; and feature engineering: normalization metrics (Min-MaxScaling) and time-series feature extraction (such as moving average load).
[0018] The algorithm selection and training in S2.2 include DQN: suitable for discrete action spaces (such as task scheduling), which improves sample efficiency through experience replay (Replay Buffer); and A3C / PPO: suitable for continuous action spaces (such as resource allocation), which supports distributed training to accelerate convergence.
[0019] The strategy deployment and online learning in S2.3 include: hierarchical reinforcement learning: processing global resource allocation and local task scheduling in layers to reduce complexity; offline pre-training: training an initial model based on historical data to simulate real load scenarios; online fine-tuning: balancing exploration and utilization through an ε-greedy strategy (initial ε=0.3, gradually decaying); and hot model update: periodically synchronizing online strategies and offline models to avoid strategy drift.
[0020] Preferably, the cache optimization in S3.1 includes hot data locking: keeping frequently accessed data in the memory cache for a long time; cache preheating: actively loading historical hot data when the system starts; asynchronous I / O: using AIO to avoid thread blocking and improve throughput;
[0021] The storage architecture optimization in S3.2 includes: SSD caching layer: adding an SSD caching layer in front of the HDD, improving random read and write performance by 100 times; hybrid storage: storing hot data on SSDs and cold data on HDDs; RAID configuration: selecting RAID0 (performance), RAID 10 (balanced), or RAID 5 / 6 (capacity) according to requirements.
[0022] The file system optimization in S3.3 includes selecting a suitable file system (such as XFS for large files and EXT4 for small files) and adjusting the I / O scheduling algorithm (such as deadline for databases and CFQ for desktop systems).
[0023] The present invention also proposes an apparatus for implementing the above method, comprising: a load monitoring unit deployed on each computing node to periodically report resource status; a scheduling decision unit with a built-in strategy engine and real-time optimization algorithm; and a cache management unit that supports LRU / LFU hybrid eviction strategies.
[0024] Preferably, the scheduling decision unit adopts a microservice architecture, which supports horizontal scaling to thousands of concurrent instances.
[0025] Preferably, it also includes: a visual monitoring panel that displays key indicators such as cluster resource utilization and task throughput; and an API gateway layer that provides standardized interfaces for external systems to call.
[0026] The present invention also proposes a computer-readable storage medium storing program code implementing the above method, wherein the code includes at least: a dynamic scheduling algorithm module; a data encoding converter module; and a caching strategy configuration module.
[0027] Preferably, the program code supports deployment on mainstream big data platforms such as Kubernetes and Spark.
[0028] 3. Beneficial effects
[0029] Compared with the prior art, the advantages of this invention are:
[0030] (1) In this invention, the problems of resource waste and performance bottleneck in big data applications are solved by dynamic resource scheduling and intelligent caching collaboration, which is especially suitable for high time-sensitive fields such as the Internet of Things and financial technology.
[0031] (2) In this invention, by reasonably designing a hierarchical caching architecture and a preloading strategy, disk I / O operations can be significantly reduced, access latency can be reduced, and the overall system performance can be improved. In actual implementation, targeted optimization is required based on specific business scenarios, data characteristics, and hardware environment. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating a big data application optimization method proposed in this invention.
[0033] Figure 2 This is a schematic diagram of the process structure for adjusting the resource allocation strategy of computing nodes proposed in this invention;
[0034] Figure 3 This is a schematic diagram of the process structure for preloading high-frequency access data proposed in this invention. Detailed Implementation
[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0036] Example 1:
[0037] Reference Figure 1 A method for optimizing big data applications includes the following steps:
[0038] S1: Real-time collection of system load metrics, including CPU load, memory load, disk I / O load, and network load. CPU load includes average load and CPU utilization; memory load includes physical memory and swap space; disk I / O load includes read / write throughput and wait queue; network load includes bandwidth, number of connections, and packet loss rate. In S1, average load is obtained by using the `uptime` or `top` command to get the process queue length over 1 / 5 / 15 minutes, including running and waiting processes for I / O; CPU utilization is obtained using the `mpstat` or `top` command, distinguishing between user mode (%us), kernel mode (%sy), idle mode (%id), and other sub-metrics.
[0039] Physical memory and swap space: Monitor total, used, cache, and swap space usage using the free command; Memory pressure: Monitor the buff / cache and available fields to determine if an OOM (Out of Memory) error is triggered;
[0040] Read / write throughput: Use iostat to view disk read / write speed (MB / s) and IOPS; Wait queue: Monitor %util (device utilization) and await (average wait time) to identify bottlenecks;
[0041] Bandwidth and connection count: Monitor traffic in real time using iftop or nload, and view TCP connection status using the ss command; Packet loss rate: Analyze network stability using ping and tcpdump.
[0042] S2: Dynamically adjust the resource allocation strategy for computing nodes based on the reinforcement learning model, specifically including the following steps: S2.1: Data acquisition and preprocessing; S2.2: Algorithm selection and training; S2.3: Strategy deployment and online learning;
[0043] S3: Employs a tiered caching mechanism to preload frequently accessed data and reduce disk I / O latency. Specifically, this includes the following steps: S3.1: Cache optimization; S3.2: Storage architecture optimization; S3.3: File system optimization.
[0044] In this embodiment, an apparatus for implementing the above method includes: a load monitoring unit deployed on each computing node to periodically report resource status; a scheduling decision unit with a built-in strategy engine and real-time optimization algorithm; and a cache management unit that supports LRU / LFU hybrid eviction policies. The scheduling decision unit adopts a microservice architecture and supports horizontal scaling to thousands of concurrent instances.
[0045] This embodiment also includes: a visual monitoring panel that displays key indicators such as cluster resource utilization and task throughput; and an API gateway layer that provides standardized interfaces for external systems to call.
[0046] In this embodiment, a computer-readable storage medium stores program code that implements the above method. The code includes at least: a dynamic scheduling algorithm module; a data encoding converter module; and a caching strategy configuration module. The program code supports deployment on mainstream big data platforms such as Kubernetes and Spark.
[0047] In this embodiment, by reasonably designing a hierarchical caching architecture and preloading strategy, disk I / O operations can be significantly reduced, access latency can be lowered, and overall system performance can be improved. In actual implementation, targeted optimization is required based on specific business scenarios, data characteristics, and hardware environment.
[0048] In this embodiment, the problems of resource waste and performance bottlenecks in big data applications are solved through dynamic resource scheduling and intelligent caching, which is especially suitable for time-sensitive fields such as the Internet of Things and fintech.
[0049] Example 2:
[0050] It has the implementation content of the above embodiments, wherein the specific implementation methods of the above embodiments can be referred to the above description, and the embodiments here will not be described in detail again; however, the difference between the embodiments in this application and the above embodiments is that:
[0051] Reference Figure 2 In this embodiment, data acquisition and preprocessing in S2.1 includes: toolchain: Prometheus for collecting system metrics, Kafka for real-time data transmission, and Spark / Flink for window aggregation; feature engineering: normalization metrics (Min-Max Scaling) and time-series feature extraction (such as moving average load).
[0052] In S2.2, algorithm selection and training include DQN: suitable for discrete action spaces (such as task scheduling), which improves sample efficiency through experience replay (Replay Buffer); and A3C / PPO: suitable for continuous action spaces (such as resource allocation), which supports distributed training to accelerate convergence.
[0053] In S2.3, policy deployment and online learning include: hierarchical reinforcement learning: handling global resource allocation and local task scheduling in layers to reduce complexity; offline pre-training: training an initial model based on historical data to simulate real load scenarios; online fine-tuning: balancing exploration and utilization through an ε-greedy policy (initial ε=0.3, gradually decaying); and hot model updates: periodically synchronizing online policies and offline models to avoid policy drift.
[0054] Example 3:
[0055] It has the implementation content of the above embodiments, wherein the specific implementation methods of the above embodiments can be referred to the above description, and the embodiments here will not be described in detail again; however, the difference between the embodiments in this application and the above embodiments is that:
[0056] Reference Figure 3 In this embodiment, the cache optimization in S3.1 includes hot data locking: keeping frequently accessed data in the memory cache for a long time; cache preheating: actively loading historical hot data when the system starts; asynchronous I / O: using AIO to avoid thread blocking and improve throughput.
[0057] Storage architecture optimizations in S3.2 include: SSD caching layer: adding an SSD caching layer in front of HDDs improves random read and write performance by 100 times; Hybrid storage: storing hot data on SSDs and cold data on HDDs; RAID configuration: choosing RAID 0 (performance), RAID 10 (balanced), or RAID 5 / 6 (capacity) according to needs.
[0058] File system optimization in S3.3 includes selecting a suitable file system (e.g., XFS is optimized for large files, and EXT4 is optimized for small files); and adjusting the I / O scheduling algorithm (e.g., deadline is suitable for databases, and CFQ is suitable for desktop systems).
[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A big data application optimization method, characterized in that, The method comprises the following steps: S1: Real-time acquisition of system load indicators, including CPU load, memory load, disk I / O load and network load, the CPU load including average load and CPU usage, the memory load including physical memory and swap area, the disk I / O load including read-write throughput and waiting queue, and the network load including bandwidth, connection number and packet loss rate; S2: Dynamically adjusting the computing node resource allocation strategy based on a reinforcement learning model, specifically comprising the following steps: S2.1: data acquisition and preprocessing; S2.2: algorithm selection and training; S2.3: strategy deployment and online learning; S3: Preloading high-frequency access data by using a hierarchical caching mechanism to reduce disk I / O delay, specifically comprising the following steps: S3.1: cache optimization; S3.2: storage architecture optimization; S3.3: file system optimization. In S1, the average load is obtained by uptime or top command within 1 / 5 / 15 minutes, including running and waiting IO processes; the CPU usage is obtained by using mpstat or top command, distinguishing user state, kernel state, idle and other subdivision indicators; 2. The big data application optimization method of claim 1, wherein, The physical memory and swap area are monitored by free command to monitor the total amount, used, cache and swap partition usage; the memory pressure is monitored by focusing on buff / cache and available fields to determine whether OOM is triggered; The read-write throughput is obtained by using iostat to view the disk read-write rate and IOPS; the waiting queue is monitored by monitoring %util and await to determine the bottleneck; The bandwidth and connection number are monitored by iftop or nload in real time, and the TCP connection state is viewed by using the ss command; the packet loss rate is analyzed by combining ping and tcpdump to analyze network stability. In S2.1, data acquisition and preprocessing include tool chain: Prometheus collects system indicators, Kafka transmits data in real time, and Spark / Flink performs window aggregation; feature engineering: normalize indicators and extract time series features; 3. The method of claim 1, wherein, In S2.2, algorithm selection and training include DQN: suitable for discrete action space, and improves sample efficiency through experience replay; A3C / PPO: suitable for continuous action space, supports distributed training to accelerate convergence; In S2.3, strategy deployment and online learning include hierarchical reinforcement learning: hierarchical processing of global resource allocation and local task scheduling to reduce complexity; strategy deployment and online learning include offline pre-training: training an initial model based on historical data to simulate real load scenarios; online fine-tuning: balancing exploration and utilization through an epsilon-greedy strategy; model hot update: periodically synchronizing online strategies and offline models to avoid strategy drift. 4. The big data application optimization method of claim 1, wherein, The cache optimizations in S3.1 include: hot data locking: keeping frequently accessed data in the memory cache for a long time; cache preheating: actively loading historical hot data when the system starts; asynchronous I / O: using AIO to avoid thread blocking and improve throughput. The storage architecture optimization in S3.2 includes: SSD caching layer: adding an SSD caching layer in front of the HDD, improving random read and write performance by 100 times; hybrid storage: storing hot data on SSDs and cold data on HDDs; RAID configuration: selecting RAID 0, RAID 10, or RAID 5 / 6 according to requirements. The file system optimization in S3.3 includes selecting a suitable file system and adjusting the I / O scheduling algorithm.
5. An apparatus for implementing the method of any one of claims 1-4, characterized by include: The load monitoring unit is deployed on each computing node and periodically reports the resource status; The scheduling decision unit has a built-in strategy engine and real-time optimization algorithm. The cache management unit supports a hybrid LRU / LFU eviction policy.
6. The apparatus of claim 5, wherein, The scheduling decision unit adopts a microservice architecture and supports horizontal scaling to thousands of concurrent instances.
7. The apparatus of claim 5, wherein, Also includes: A visual monitoring panel displays key metrics such as cluster resource utilization and task throughput. The API gateway layer provides standardized interfaces for external systems to call.
8. A computer-readable storage medium, characterized in that, The system stores program code that implements the method of any one of claims 1-7, the code comprising at least: a dynamic scheduling algorithm module; a data encoding converter module; and a caching strategy configuration module.
9. The storage medium of claim 8, wherein, The program code supports deployment on mainstream big data platforms such as Kubernetes and Spark.