A storage device capacity and performance collection method and system

Abstract

The application provides a storage device capacity and performance acquisition method and system. The method comprises the following steps: a crawling program is used to crawl running log information of the storage device; the crawling program determines the capacity of the storage device according to the running log information; the storage device is analyzed and researched for I / O performance evaluation by using a pre-set storage system I / O performance analysis model according to request service information in the running log; and therefore, the running data of the storage device can be quickly acquired by using the crawling program, the capacity of the storage device and the I / O performance evaluation can be determined, and the method has wide adaptability and low cost.

Description

Technical Field

[0001] This invention belongs to the field of storage capacity setting technology, and more specifically, relates to a method and system for acquiring storage device capacity and performance. Background Technology

[0002] Faced with the rapidly increasing data storage volume across all industries, and with the continuous increase in storage devices, the maintenance of these devices consumes a lot of manpower and funds. Furthermore, the risk of human error increases due to differences in operating methods and procedures for different devices.

[0003] In existing storage capacity determination technologies, a physical page query request for the target storage unit is sent to the target storage system, and the number of physical pages and physical page capacity corresponding to the target storage unit, determined based on the query identifier information, are received from the target storage system. The capacity corresponding to the target storage unit is then determined based on the number of physical pages and physical page capacity.

[0004] The existing technology mainly analyzes some local features, including runtime features based on I / O load such as I / O request arrival rate, I / O request size, read / write percentage, access ordering, and burstiness; and system performance including throughput, bandwidth, and I / O response time. The storage system performance model extraction method uses multiple regression theory to fit the characteristic parameters of I / O load and the average response time of the system to extract the performance model of the storage system online.

[0005] However, this existing technology is accurate and effective in storage system environments with relatively simple functions and structures. But its implementation cost is high, and the scale and complexity of current mainstream storage systems have revealed significant limitations and shortcomings in terms of accuracy, practicality, and effectiveness. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for acquiring storage device capacity and performance, which is used to determine the capacity and I / O performance evaluation of storage devices, and has wide applicability and low cost.

[0007] The first aspect of this application discloses a method for acquiring storage device capacity and performance, including:

[0008] Use a web crawler to crawl the operating log information of the storage device;

[0009] The crawling program determines the capacity of the storage device based on the running log information;

[0010] Based on the request service information in the operation log, the storage device is analyzed and its I / O performance is evaluated using a pre-set storage system I / O performance analysis model.

[0011] Optionally, the operation log information of the storage device can be crawled using a crawling program, including:

[0012] Based on the type of storage device, obtain a pre-set query request;

[0013] Use a crawler to send a query request to the storage device;

[0014] The storage device obtains its own operation log information based on the identification information in the query request and feeds it back to the crawling program.

[0015] Optionally, the calculation formula used to determine the capacity of the storage device is:

[0016] The formula for bit-by-bit calculation is: Storage capacity = Number of storage units x Length of storage word;

[0017] The formula for calculating storage capacity per byte is: Storage capacity = Number of storage units x Storage word length / 8.

[0018] Optionally, the performance of the storage device is reflected by the performance of the nodes, which is reflected by the number of requests waiting in the cache and I / O servers and the number of I / O server failures.

[0019] Optional, the process of establishing a storage system I / O performance analysis model includes:

[0020] Set up a queuing system for the computer; wherein the queuing system takes the form of: representing the time difference distribution of requests arriving successively; representing the distribution of server service time; representing the number of computer servers; representing the computer's capacity; representing the number of requests; and representing the rules for the computer to provide services.

[0021] Optionally, a preset negative exponential distribution and corresponding parameters can be substituted into the queuing system;

[0022] Wherein, the negative exponential distribution of parameter λ represents the distribution of request arrival time intervals, 0 < λ < ∞; there is k positions at the front end, 0 < k < ∞; if there is s I / O server, 0 < s ≤ k < ∞; there are ks cache spaces, 0 < ks < ∞; the request source is ∞; and the service time follows the parameter μ. i The negative exponential distribution of (i = 1, 2), 0 < μ1 < μ2 < ∞;

[0023] When the waiting queue length is greater than 0, the service time follows the parameter; when the waiting queue length is less than or equal to 0, the service time follows the parameter, and the first-come-first-served rule is adhered to.

[0024] Optionally, the storage system I / O performance analysis model reflects the dynamic characteristics of node performance through state transitions; and obtains the average number of requests, the average number of failed servers, the ratio of requests encountering full queues, and the total request loss rate.

[0025] Optionally, the formula used for the average number of requests is:

[0026] The formula used for the average number of failed servers is:

[0027] The formula used to determine the ratio of requests encountering a full queue is:

[0028] The formula used for the overall request loss rate is:

[0029] Let N(t) = n represent the number of requests waiting in the cache and lingering in the I / O server at time t, where 0 ≤ n < k; M(t) = m represent the number of I / O server failures at time t, with k positions at the front end, 0 < k < ∞; and s I / O servers; Lm is the average number of requests; E[D] is the average number of failed servers; PB is the ratio of requests encountering a full queue; PL is the total request loss rate; ν is the system transfer rate; L Q This represents the average number of requests waiting in the front end.

[0030] The second aspect of this application discloses a storage device capacity and performance acquisition system, characterized in that it includes:

[0031] The crawling unit is used to crawl the operation log information of the storage device using a crawling program;

[0032] A capacity unit is defined, which is used to determine the capacity of the storage device based on the running log information using the crawling program;

[0033] The performance evaluation unit is used to analyze and study the I / O performance of the storage device based on the request service information in the operation log and through the storage system I / O performance analysis model.

[0034] Optionally, when the crawling unit uses a crawling program to crawl the operation log information of the storage device, it is specifically used for:

[0035] Based on the type of storage device, obtain a pre-set query request;

[0036] Use a crawler to send a query request to the storage device;

[0037] The storage device obtains its own operation log information based on the identification information in the query request and feeds it back to the crawling program.

[0038] As can be seen from the above technical solution, the storage device capacity and performance acquisition method provided by the present invention includes: using a crawling program to crawl the operation log information of the storage device; the crawling program determining the capacity of the storage device based on the operation log information; and analyzing and studying the I / O performance evaluation of the storage device based on the request service information in the operation log through a pre-set storage system I / O performance analysis model. Thus, the crawling program can quickly obtain the operation data of the storage device, thereby determining the capacity and I / O performance evaluation of the storage device. It has wide applicability and low cost. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a method for collecting storage device capacity and performance according to an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of the data crawling process in a method for collecting storage device capacity and performance data provided in an embodiment of the present invention;

[0042] Figure 3 This is a queuing flowchart of a queuing system in a method for collecting storage capacity and performance data provided in an embodiment of the present invention;

[0043] Figure 4 This is a queuing flowchart of the queuing system in another method for collecting storage device capacity and performance provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of a storage device capacity and performance acquisition system provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] It should be noted that the terms used in this application are interpreted as follows:

[0048] Storage capacity: Storage capacity refers to the total number of bits of binary code that main memory can store.

[0049] IOPS (Input / Output Per Second): This refers to the number of input / output requests (or read / write operations) per second, and is one of the main metrics for measuring storage media performance. IOPS indicates the number of read / write requests the system can handle per second.

[0050] Storage unit: The smallest unit of memory accessed by the CPU; each storage unit has an address.

[0051] Storage word length: The number of bits of binary code stored in a single storage unit (storage address) in memory.

[0052] This application provides a method for collecting storage device capacity and performance data. This method is accurate and effective in storage systems with relatively simple functions and structures, but has a high implementation cost. Furthermore, the scale and complexity of current mainstream storage systems have exposed significant limitations and shortcomings in terms of accuracy, practicality, and effectiveness.

[0053] See Figure 1 The method for collecting the capacity and performance of the storage device includes:

[0054] S101. Use a crawling program to crawl the operation log information of the storage device.

[0055] It's important to note that with the advancements in computer technology, data storage, usage, and security are crucial factors for business profitability and future development. Consequently, companies invest significant resources—material, human, and financial—in improving these areas. As a vital resource in the information age, storage systems possess substantial practical value. The explosive growth of digital information, coupled with a large user base and diverse applications, has led to a demand for high-capacity information storage systems. Developing storage systems is the preferred and common approach to facilitate the use of numerous resources; therefore, timely access to key information such as the capacity and performance of storage systems is an essential technical requirement.

[0056] Performance modeling of storage systems has always been a hot research topic in the storage field. Recent research can be categorized into three main areas. The first is based on analytical methods to build performance models of storage systems, such as those for disk array systems. These studies strictly adhere to the internal operating principles of disk arrays. However, these models generally require prior knowledge of the storage system; that is, their creation necessitates understanding its internal workings. Furthermore, changes in system configuration, such as software upgrades, render the original performance model meaningless, resulting in poor portability. The second approach uses measurement methods, employing table-based built-in methods to model disk array systems. This method is simple and effective, but requires extensive and complex prior measurements, leading to low efficiency. The third approach uses a black-box method, treating the disk array system as a black box and employing statistical inference to evaluate the response time of a single I / O request.

[0057] S102. The crawling program determines the capacity of the storage device based on the running log information.

[0058] It should be noted that the operation log information includes operation data, so the capacity of the storage device can be determined by the operation log information.

[0059] For example, if the rated capacity of a storage device is fixed, the capacity of the storage device can be obtained by subtracting the operating data capacity from the rated capacity. Of course, this is merely an example; other implementation methods are not detailed here, but are all within the scope of this application.

[0060] S103. Based on the request service information in the operation log, analyze and study the I / O performance evaluation of the storage device through the pre-set storage system I / O performance analysis model.

[0061] In other words, the requested service information serves as the input to the storage system's I / O performance analysis model, while the I / O performance evaluation serves as the output of the storage system's I / O performance analysis model.

[0062] In this embodiment, the crawling program can quickly obtain the operating data of the storage device, thereby determining the storage device's capacity and I / O performance evaluation. It has wide applicability and low cost.

[0063] In practical applications, see Figure 2 Step S101, the specific process of using a crawling program to crawl the operation log information of the storage device is as follows:

[0064] (1) Obtain a pre-set query request based on the type of storage device.

[0065] In other words, different types of storage devices require different query requests; therefore, it is necessary to obtain the corresponding query requests for different storage types.

[0066] (2) Use a crawler to send a query request to the storage device.

[0067] It should be noted that the data can be sent via a communication connection or wirelessly; no specific limitation is made here, and the choice depends on the actual situation, both of which are within the scope of protection of this application.

[0068] (3) The storage device obtains its own operation log information based on the identification information in the query request and feeds it back to the crawling program.

[0069] In other words, the crawling program sends query requests to all storage devices, and the query requests carry the identification information of the corresponding storage devices. After receiving the query request, each storage device can first determine whether the identification information in the query request matches its own identification information. If they match, it can obtain its own operation log information and feed it back to the crawling program.

[0070] It should be noted that the web crawler can be Python. Of course, the web crawler is not limited to this method; other methods will not be elaborated here, and the appropriate method can be chosen depending on the actual situation, all of which are within the scope of protection of this application.

[0071] Specifically, the process first retrieves a pre-defined query request based on the storage device type, then sends the query request to the storage device using Python. The target storage device retrieves its own relevant information based on the query identifier and sends it back to Python. See the detailed process below. Figure 3 As shown.

[0072] In practical applications, the formula used to determine the capacity of a storage device is:

[0073] The formula for bitwise calculation is: Storage capacity = number of storage cells x storage word length.

[0074] The formula for calculating storage capacity per byte is: Storage capacity = Number of storage units x Storage word length / 8.

[0075] In other words, either a formula for bitwise calculation or a formula for byte-wise calculation can be selected, and it is also possible to combine the two formulas; no specific limitation is made here, it depends on the actual situation, and all are within the scope of protection of this application.

[0076] It's important to note that the data from the front-end's access to the servers in the storage system is detectable; the data for each request and service can be obtained from the runtime logs. By monitoring the service information for each request from each front-end server, the performance of the entire storage system can be analyzed. Many factors can reflect or influence the performance of a storage system, but the I / O performance of the storage system when serving each front-end request more fully reflects its overall performance.

[0077] In practical applications, the performance of storage devices is reflected through the performance of nodes, which in turn is reflected by the number of requests waiting in the cache and I / O servers, and the number of I / O server failures.

[0078] In other words, the more requests waiting in the I / O server cache, the worse the storage speed of the storage system, or its storage performance. Clearly, the more I / O server failures, the worse the I / O performance of the storage system.

[0079] It's important to note that the scope of the computer's front-end operation is limited; that is, the capacity of the front end is finite, meaning the number of waiting positions in the queue is limited. Furthermore, if the waiting time for a request at the front end exceeds a certain threshold, the request will be abandoned. Based on these characteristics, we established a repairable queuing model using queuing theory, incorporating variable failure rates, timeouts, and variable service rates. The specific process is detailed below. Figure 3 .

[0080] In other words, when a request arrives at the front end, it is queued. If the queue is full or the wait times out, the request is abandoned or re-queued. When the request begins to be processed, it is dispatched to the appropriate I / O server.

[0081] In practical applications, the process of establishing a storage system I / O performance analysis model includes:

[0082] Configure a computer queuing system.

[0083] It should be noted that computer queuing systems are specifically categorized based on computer information input, queuing patterns, and service rules. There are many queuing methods that can be categorized, and this classification method is beneficial for representing various queuing patterns. The form of this queuing system is X / Y / Z / A / B / C; X represents the time difference distribution of requests arriving successively; Y represents the distribution of server service time; Z represents the number of computer servers; A represents the computer's capacity; B represents the number of requests; and C represents the rules for the computer to provide services.

[0084] In this sequence, X / Y / Z / A / B / C have decreasing priority from left to right. Specifically, they can be queued strictly according to the above format, or the parameters can be weighted according to the priority format to obtain the final queue sequence. Specific queuing methods will not be elaborated here, as they are all within the scope of protection of this application.

[0085] It should be noted that the classic queuing method assumes that the computer service system will not experience any unexpected problems. However, in actual operation, unexpected problems often occur, leading to delays in providing services to requests. In such cases, maintenance personnel need to repair the computer service system. After the repair is completed, the request can be submitted again. A queuing system where the computer can recover from unexpected problems is called a repairable queuing model. The queuing index values ​​that need to be analyzed and studied in a repairable queuing model must fully consider the following factors:

[0086] (1) Queue length and waiting queue length: The former is the number of requests in the system, and the latter is the number of requests waiting. The number of requests in the system and the number of requests waiting for service are not regular. They are activities based on individual needs. Therefore, they are important quantitative indicators in the process of requesting and serving.

[0087] (2) Request waiting time: Requesting a service requires waiting for a period of time before the service can be enjoyed. This period of time is the computer's request waiting time.

[0088] (3) Busy period and idle period: The busy period of computer operation refers to the time from when the request reaches the idle system state until the system becomes idle again. The idle period of computer system refers to the time when the computer does not request service. In contrast to the busy period, the system is in an idle state for a long time.

[0089] Service equipment in a repairable queuing system is prone to malfunctions, which may result in the inability to provide timely service to users or the termination of ongoing services, thereby affecting server performance. Therefore, the reliability of such a queuing system is an issue.

[0090] For repairable queuing systems, they are categorized based on the number of service devices that experienced an accident:

[0091] 1) A completely unrepairable queuing system, which means that if all the equipment providing the computer service fails, the entire service equipment cannot operate.

[0092] 2) A feasible and repairable queuing system, meaning that after a computer service equipment malfunctions, not all equipment will fail; damaged equipment can continue to operate by relying on intact equipment, maintaining the normal operation of the entire service system. See the new queuing model. Figure 4 .

[0093] In other words, a preset negative exponential distribution and corresponding parameters are substituted into the queuing system; at the same time, if a timeout occurs or the position is full, the system is abandoned directly without asking for a retry.

[0094] Here, the negative exponential distribution with parameter λ represents the distribution of request arrival time intervals, 0 < λ < ∞; there are k front-end locations, 0 < k < ∞; there are s I / O servers, 0 < s ≤ k < ∞; there are ks cache spaces, 0 < ks < ∞; the request source is ∞; and the service time follows a parameter μ. i The negative exponential distribution of (i = 1, 2) is 0 < μ1 < μ2 < ∞.

[0095] When the waiting queue length is greater than 0, the service time follows parameter μ2; when the waiting queue length is less than or equal to 0, the service time follows parameter μ1, adhering to the first-come, first-served rule.

[0096] This paper primarily analyzes and studies I / O performance evaluation by combining queuing systems and quasi-birth-death processes. The content mainly includes I / O service devices, structural layout, and operational modes. The most basic operational step is as follows: when any request arrives at the front end, it will be retried or abandoned if the front end is full or the waiting timeout occurs. If the front end is not full, the request enters the front end. In the front end, if there is no waiting and there is free space, it will be served directly. When a request times out, it may be abandoned, or abandoned and then retried after a certain period of time. The quasi-birth-death process is an extension of the traditional standard birth-death process. This part combines queuing theory and quasi-birth-death processes to build a model, using the characteristics of the infinitesimal generator matrix (Q-matrix) of the quasi-birth-death process to analyze the I / O problem and derive corresponding prediction results and related performance indicators.

[0097] In practical applications, in order to further evaluate the performance of the storage system and determine its performance index values, the dynamic performance can be reflected by the state model of the storage I / O system, and a solution algorithm for the model can be proposed.

[0098] Let N(t) = n represent the number of requests waiting in the cache and lingering on the I / O server at time t, where 0 ≤ n < k, and M(t) = m represent the number of I / O server failures at time t. The negative exponential distribution is characterized by mutual independence and memorylessness, and the allowed waiting time limit for requests is based on this distribution. Therefore, the time interval during which requests are not allowed to be abandoned follows a negative exponential distribution.

[0099] Let X(t) = (M(t), N(t)) represent the state of the system at time t, and let it have a state space. Here, state space is a term from control engineering. A state refers to the smallest ordered set of variables that determine the state of a system. The state space, then, refers to the set of all possible states of the system.

[0100] S={m,n,0≤m≤s,0≤n≤k}.

[0101] The fundamental premise is that both the allowed waiting time limit and the time interval during which requests cannot be abandoned follow a negative exponential distribution. Service time, request arrival interval, I / O server failure interval, repair time, and patient waiting time all conform to this premise. Therefore, from a state-space perspective, X(t) is a Markov process. Because the state space of X(t) is finite and irreducible, the stationary distribution of X(t) is unique.

[0102] Specifically, the dynamic characteristics of node performance are reflected through state transitions.

[0103] Let ∏ = (π0, π1, π2, ..., π) s ), where π m =(π) m0 ,π m1 ,π m2 ,…,π mk (0≤m≤s), which is the state transition equation. It directly applies the stationary distribution of X(t) and uses the model description as the basis to transition from state (m,n) to each state.

[0104] π m =π0R m ,(0≤m≤s);

[0105] In the formula, R0 = I, and I is the identity matrix;

[0106]

[0107] π0 is π0(R) s-1 C s-1 +R s A s A basic solution of ) = 0;

[0108] Furthermore, because the stationary distribution of X(t) satisfies

[0109] Then π m C m +π m+1 A m+1 +π m+2 B m+2 =0,π s-2 C s-2 +π s-1 A s-1 +π s B s =0;

[0110] And because of B m (1≤m≤s) is invertible, therefore

[0111] By analogy and iterative steps, we can obtain:

[0112] π m =π0R m ,in

[0113] Let R0 = I, and substituting it into the second equation for X(t), we can obtain e is a k+1 order column vector of all 1s, so π0 can be solved.

[0114] Where π0-πs represents the stationary distribution of X(t); R0-R m R s 、,A0-A m+1 A s C0-C m ,;B0-B m+1 B s Both and Q are parts of the expression in the model solution and have no practical meaning.

[0115] We can use the above formulas to calculate the performance of some related storage devices.

[0116] In other words, the storage system I / O performance analysis model reflects the dynamic characteristics of node performance through state transitions; it obtains the average number of requests, the average number of failed servers, the ratio of requests encountering full queues, and the total request loss rate.

[0117] In practical applications, the formula used for average number of requests is:

[0118] The formula used for mean time between failures (MTBF) servers is:

[0119] The formula used to determine the ratio when a request encounters a full queue is:

[0120] The formula used for the overall request loss rate is:

[0121] Where N(t) = n represents the number of requests waiting in the cache and lingering in the I / O server at time t, 0 ≤ n < k; M(t) = m represents the number of I / O server failures at time t, with k positions at the front end, 0 < k < ∞; and s I / O servers; Lm is the average number of requests; E[D] is the average number of failed servers; PB is the ratio of requests encountering a full queue; PL is the total request loss rate; ν is the system transfer rate; L Q This represents the average number of requests waiting in the front end.

[0122] In this embodiment, I / O performance was analyzed using a combination of queuing theory and quasi-birth-death processes. Specific performance index expressions for I / O operation were summarized, along with operational rules under different I / O conditions and general rules for all operating modes. Performance evaluation methods primarily include mathematical models, analytical methods, and simulation methods. Generally, mathematical models and analytical methods are mainly used in the first half of the computer system configuration process. The mathematical model is based on queuing theory, while analytical methods summarize the computer's operating process. This research method requires minimal workload, saves research time, and can yield preliminary research conclusions. Assuming the established model matches reality, the future direction of computer development can be predicted.

[0123] Another embodiment of this application provides a storage device capacity and performance acquisition system.

[0124] See Figure 5 The storage device capacity and performance acquisition system includes:

[0125] The crawling unit 101 is used to crawl the operation log information of the storage device using a crawling program.

[0126] Capacity unit 102 is used to determine the capacity of the storage device based on the running log information using a crawling program.

[0127] The performance evaluation unit 103 is used to analyze and study the I / O performance of the storage device based on the request service information in the operation log and through the storage system I / O performance analysis model.

[0128] In practical applications, when the crawling unit 101 is used to crawl the operation log information of the storage device using a crawling program, it is specifically used for:

[0129] Based on the type of storage device, retrieve the pre-set query request.

[0130] Use a crawler to send query requests to the storage device.

[0131] The storage device retrieves its own operating log information based on the identification information in the query request and feeds it back to the crawling program.

[0132] For details on the specific working process and principle of each of the above units, please refer to the storage device capacity and performance acquisition method provided in the corresponding embodiments above. They will not be repeated here, and are all within the protection scope of this application.

[0133] The features described in the various embodiments of this specification can be substituted for or combined with each other. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0134] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for collecting storage device capacity and performance, characterized by, include: Use a web crawler to crawl the operating log information of the storage device; The crawling program determines the capacity of the storage device based on the running log information; Based on the request service information in the operation log, the storage device is analyzed and its I / O performance is evaluated using a pre-set storage system I / O performance analysis model. The storage system I / O performance analysis model reflects the dynamic characteristics of node performance through state transitions. The average number of requests, the average number of failed servers, the ratio of requests encountering full queues, and the total request loss rate are obtained. The formula employed for the average number of requests is: ; The formula employed for the average number of failed servers is: ; The formula employed for the ratio of requests that encounter a queue full is: ; The total request loss rate employs a formula as follows: ; Among them, let Indicates time The number of requests waiting in the cache and lingering on the I / O server is , , Indicates time The number of I / O server failures, front end has One location, ;have I / O servers; Lm is the average number of requests, E[D] is the average number of failed servers; PB is the ratio of requests encountering a full queue; PL is the total request loss rate; ν is the system transfer rate; L Q This represents the average number of requests waiting in the front end.

2. The method of claim 1, wherein, The operation log information of the storage device is crawled using a crawling program, including: Based on the type of storage device, obtain a pre-set query request; Use a crawler to send a query request to the storage device; The storage device obtains its own operation log information based on the identification information in the query request and feeds it back to the crawling program.

3. The method of claim 1, wherein, The formula used to determine the capacity of the storage device is as follows: The formula for bitwise calculation is: Storage capacity = number of storage units x storage word length; The formula for calculating storage capacity per byte is: Storage capacity = Number of storage units x Storage word length / 8.

4. The method of claim 1, wherein, The performance of the storage device is reflected in the performance of the nodes, which is reflected in the number of requests waiting in the cache and I / O servers, and the number of I / O server failures.

5. The method of claim 4, wherein, The process of establishing a storage system I / O performance analysis model includes: Set up a computer queuing system; Wherein, the queuing system is in the form of ; representing the time difference distribution of successive requests; representing the distribution of server service time; representing the number of computer servers; representing the capacity of the computer; representing the number of requests; representing the rules of computer service.

6. The method of claim 5, wherein, The queuing system is substituted with a preset negative exponential distribution and corresponding parameters; where the parameter is The negative exponential distribution with parameter ; there are positions in the front end, ; if there are I / O servers, ; there are cache spaces, ; the request source is ; the service time obeys the negative exponential distribution with parameter , ; Use the service time compliance parameter when the waiting queue length is greater than 0 Use the service time compliance parameter when the waiting queue length is less than or equal to 0 , adhering to the first-come-first-served rule.

7. A storage device capacity and performance collection system, comprising: include: The crawling unit is used to crawl the operation log information of the storage device using a crawling program; A capacity unit is defined, which is used to determine the capacity of the storage device based on the running log information using the crawling program; The performance evaluation unit is used to analyze and study the I / O performance of the storage device based on the request service information in the operation log and through the storage system I / O performance analysis model. The storage system I / O performance analysis model reflects the dynamic characteristics of node performance through state transitions. The average number of requests, the average number of failed servers, the ratio of requests encountering full queues, and the total request loss rate are obtained. The formula employed for the average number of requests is: ; The formula employed for the average number of failed servers is: ; The formula employed for the ratio of requests that encounter a queue full is: ; The total request loss rate employs a formula as follows: ; Among them, let Indicates time The number of requests waiting in the cache and lingering on the I / O server is , , Indicates time The number of I / O server failures, front end has One location, ;have I / O servers; Lm is the average number of requests, E[D] is the average number of failed servers; PB is the ratio of requests encountering a full queue; PL is the total request loss rate; ν is the system transfer rate; L Q This represents the average number of requests waiting in the front end.

8. The storage device capacity and performance collection system of claim 7, wherein, When the crawling unit is used to crawl the operation log information of the storage device using a crawling program, it is specifically used for: Based on the type of storage device, obtain a pre-set query request; Use a crawler to send a query request to the storage device; The storage device obtains its own operation log information based on the identification information in the query request and feeds it back to the crawling program.

Images