A method for balancing system resources and disk real-time performance and a readable storage medium
By balancing the data to different threads in network statistics and dynamically adjusting the block size, the problems of resource jitter and data drop-in in the existing technology are solved, and the effects of stable utilization of system resources and real-time data drop-in are achieved.
Patent Information
- Application Number
- CN202011590935.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-29
AI Technical Summary
The existing technology cannot dynamically adjust memory resources when facing large data volumes, resulting in serious jitter between CPU and IO resources, and the memory block cannot be shrinked in time when there is a small amount of traffic, affecting system performance and real-time data entry.
By allocating data to different threads, setting a time refresh mechanism for hard and soft limits, dynamically adjusting the block size and refresh priority, ensuring smooth utilization of system resources and real-time data drop.
Avoid resource peaks in a jitter traffic environment, maintain stable utilization of CPU and IO, and ensure real-time data entry and query delay control.
Smart Images

Figure CN112650589B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data storage and retrieval, and in particular to a method for balancing system resources and disk real-time performance and a readable storage medium. Background Art
[0002] In network statistics projects, it is often necessary to query the information generated by IP in a large data volume environment. The efficiency of the query depends largely on the storage, which involves the balance of CPU, IO, memory and query real-time performance.
[0003] Current common technical solutions:
[0004] Currently, many common refresh solutions integrate and write cache blocks to disk when they reach a certain memory limit. Generally, the background thread is used to directly refresh data to disk, or perform some merge operations, which increases the CPU and IO load. System resources CPU, IO and memory are prone to peak values, thus affecting the overall performance of the system.
[0005] The existing technology has a very obvious disadvantage. When faced with sudden large amounts of data, the control of memory is not precise enough and cannot be scaled up and down dynamically, which will cause serious jitters in the system resources CPU and IO at a certain point in time. At the same time, when there is a small amount of traffic, the memory block cannot be dynamically shrunk in time, so as to make the idle CPU and IO work as evenly as possible, that is, to make the data fall to the disk as much as possible, to ensure the real-time data fall to the disk, and at the same time reduce the subsequent data accumulation when facing sudden large traffic, which will cause excessive load on the CPU and IO and affect the system throughput. Summary of the invention
[0006] The purpose of the present invention is to provide a method for balancing system resources and disk real-time performance in order to overcome the defects of the prior art. The method of the present invention balances the use of system resources, minimizes the peak value of system resources, and ensures that data is stored on disk to improve the real-time performance of queries.
[0007] The object of the present invention is achieved through the following technical solutions:
[0008] On the one hand, the present invention discloses a method for balancing system resources and real-time performance of disk storage, the method comprising the following steps: S1: evenly distributing different data in different threads, hashing IP flows and distributing them, and dividing them into several blocks in each thread; S2: setting the total refresh cache value of each thread and the maximum refresh cache value of each block of the thread according to the system's memory and IO bandwidth, and dynamically selecting blocks for refresh and adjusting block size during system operation.
[0009] According to a preferred embodiment, the system described in step S2 is set with two time limits, namely a hard limit and a soft limit; when the Block reaches the hard time limit, the Block data is flushed to the disk; when the Block reaches the soft time limit, the Block's current cache value reaches half of the maximum refresh cache value set by the Block and is refreshed.
[0010] According to a preferred implementation, each thread can also perform flag setting of the Block under the priority refresh thread based on demand.
[0011] According to a preferred embodiment, when the flag is set on a thread, the system will select the remaining blocks in the current thread for refresh without exceeding the total refresh cache value of each thread.
[0012] According to a preferred embodiment, when hotspot traffic is encountered and concentrated on a hotspot thread, the total refresh cache value of the hotspot thread and the refresh cache values of each Block under the hotspot thread are dynamically increased.
[0013] According to a preferred embodiment, when the hotspot traffic tends to be stable, the total refresh cache value of the hotspot thread and the refresh cache value of each Block under the thread are dynamically reduced.
[0014] On the other hand, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for balancing system resources and real-time disk storage is implemented.
[0015] The aforementioned main scheme of the present invention and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection by the present invention. After understanding the scheme of the present invention, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present invention, and they are not exhaustively listed here.
[0016] Beneficial effects of the present invention: The method of the present invention can avoid the peak of system resources in a jittery traffic environment, make the CPU and IO as smooth as possible under the condition of minimum memory usage, and ensure the real-time performance of data storage. Compared with the existing technology, the method of the present invention has more stable CPU and IO utilization in a jittery traffic environment, and the real-time performance of data storage can be guaranteed, and the query delay is also controllable. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a structural diagram of allocating IP flows in the method for balancing system resources and real-time performance of disk downloading of the present invention. DETAILED DESCRIPTION
[0018] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0019] It should be noted that in order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.
[0020] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] Embodiment 1:
[0022] refer to Figure 1 As shown, the present invention discloses a method for balancing system resources and real-time performance of data storage. The method of the present invention balances the utilization of CPU, IO and memory and the real-time performance of query. After balancing CPU and IO, the total throughput of the system can be greatly improved. At the same time, the solution can ensure the real-time performance of data storage, thereby ensuring the real-time performance of query.
[0023] The method of the present invention comprises:
[0024] Step 1: Evenly distribute different data to different threads, hash the IP flow and distribute it, and divide it into several blocks in each thread. By setting multiple blocks, further subdividing data resources is achieved. Through the data equalization in the first step, the amount of data processed by each thread is basically the same.
[0025] Step 2: Set the total refresh cache value of each thread and the maximum refresh cache value of each block of the thread according to the system's memory and IO bandwidth, and dynamically select blocks for refresh and adjust block size during system operation.
[0026] Preferably, in the second step, the system is set with two time limits, namely a hard limit and a soft limit. In addition, the system is driven by seconds.
[0027] Furthermore, when the Block reaches the hard time limit, the Block data is flushed to disk. When the Block reaches the soft time limit, the Block data is flushed when the current cache value of the Block reaches half of the maximum refresh cache value set by the Block.
[0028] Preferably, each thread can also set a flag for refreshing the blocks under the thread in priority based on demand. Further, when the flag is set on the thread, the system will select the remaining blocks in the current thread for refreshing without exceeding the total refresh cache value of each thread.
[0029] Preferably, when hotspot traffic is encountered and concentrated on a hotspot thread, the total refresh cache value of the hotspot thread and the refresh cache values of each Block under the hotspot thread are dynamically increased.
[0030] Furthermore, when the hotspot traffic tends to be stable, the total refresh cache value of the hotspot thread and the refresh cache value of each Block under the thread are dynamically reduced, thereby making the system more stable.
[0031] The method of the present invention can avoid the peak of system resources in a jittery traffic environment, make the CPU and IO as smooth as possible while using the least memory, and ensure the real-time performance of data storage. Compared with the existing technology, the method of the present invention has a more stable CPU and IO utilization rate in a jittery traffic environment, and the real-time performance of data storage can also be guaranteed, and the query delay is also controllable.
[0032] Example 2
[0033] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for balancing system resources and real-time disk placement described in the above embodiment 1 is implemented.
[0034] The above basic examples of the present invention and their further selected examples can be freely combined to form multiple embodiments, all of which are embodiments that can be adopted and claimed by the present invention. In the scheme of the present invention, each selected example can be arbitrarily combined with any other basic example and selected example. Those skilled in the art will know that there are many combinations.
[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for balancing system resources and disk real-time performance, characterized in that: The method comprises the following steps: S1: Evenly distribute different data to different threads, hash the IP flow and distribute it, and divide it into several blocks in each thread; S2: Set the total refresh cache value of each thread and the maximum refresh cache value of each block of the thread according to the system's memory and IO bandwidth, and dynamically select blocks for refresh and adjust the block size during system operation; The system described in step S2 is set with two time limits, namely a hard limit and a soft limit; When the block reaches the hard time limit, the block data is flushed to the disk; When the Block reaches the soft time limit, the Block's current cache value reaches half of the maximum refresh cache value set by the Block and is refreshed; Each thread can also set the mark of the Block under the priority refresh thread based on demand; When the flag is set on a thread, the system will select the remaining blocks in the current thread for refresh without exceeding the total refresh cache value of each thread; When hot traffic is encountered and concentrated on a hot thread, the total refresh cache value of the hot thread and the refresh cache value of each block under the hot thread are dynamically increased; When the hotspot traffic tends to be stable, the total refresh cache value of the hotspot thread and the refresh cache value of each Block under the thread are dynamically reduced.
2. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for balancing system resources and real-time disk downloading as claimed in claim 1 is implemented.
