Analysis method and device for block storage write-in data and storage medium
Through the multi-dimensional data analysis method, the complexity of block storage writing data is solved, the performance and stability of the storage system are improved, resource utilization is optimized, and it is suitable for block storage systems of different scales.
Patent Information
- Application Number
- CN202510523423.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-08
AI Technical Summary
The prior art cannot comprehensively analyze the multi-dimensional characteristics of block storage written data, and it is difficult to adapt to the rapidly developing block storage technology requirements, resulting in difficulty in performance optimization and troubleshooting.
A multi-dimensional data analysis method is adopted, including comprehensive analysis of time, space, data continuity and data size distribution dimensions. The amount and number of data written are counted through segmentation period, storage area and offset, and combined with visual display results.
It realizes efficient performance optimization of block storage systems, identify performance bottlenecks and potential failures, improves system stability and resource utilization, and reduces operating costs.
Smart Images
Figure CN120447833A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of block storage technology, and in particular to an analysis method, device, and storage medium for block storage write data. Background Art
[0002] In the field of block storage technology, multi-dimensional and in-depth analysis of written data is the core link for optimizing storage system performance and improving data writing efficiency.
[0003] The existing analysis method for written data is:
[0004] The system uses the number and size of data written within a time slice as basic data. This basic data is then continuously displayed through a visual interface on multiple levels, including hard drives, storage volumes, storage pools, and systems, to show IOPS (Input / Output Operations Per Second) and bandwidth information.
[0005] The above analysis method can only obtain IOPS and bandwidth information, which is difficult to adapt to the requirements of the increasingly rapidly developing block storage technology field. Summary of the Invention
[0006] In response to the defects in the existing technology, the technical problem solved by the present invention is: how to analyze the write data of block storage from the dimensions of data fluctuation at different times, data density, write continuity, and data size distribution.
[0007] To achieve the above objectives, in a first aspect, an embodiment of the present application provides a method for analyzing block storage write data, the method comprising the following steps:
[0008] Divide the written data into fixed cycles and count the amount and number of written data in each cycle;
[0009] Divide the space for storing written data into several storage areas according to the preset storage length, and count the amount of written data and the number of write times in each storage area;
[0010] Determine the offset of adjacent written data and count the amount of written data and the number of write times at the same offset.
[0011] In combination with the first aspect, in one embodiment, the process of counting the write amount and write times of write data in each cycle includes: after obtaining the write amount and write times of write data in each cycle, determining the write amount of write data in each cycle, and the write amount ratio to the total write amount of all write data; and the write times of write data in each cycle, and the write times ratio to the total write times of all write data; using the write amount ratio and write times ratio of each cycle as the analysis result of the cycle.
[0012] In combination with the first aspect, in one embodiment, the process of counting the amount of write data and the number of write times in each storage area includes: after obtaining the amount of write data and the number of write times in each storage area, determining the amount of write data in each storage area and the ratio of the total amount of write data of all write data; and the ratio of the number of write times of write data in each storage area and the total number of write times of all write data; and using the ratio of the amount of write data and the ratio of the number of write times of each storage area as the analysis result of the storage area.
[0013] In combination with the first aspect, in one embodiment, the process of determining the offset of adjacent write data and counting the write amount and write count of write data at the same offset includes: sorting the write data in a specified order, and counting the offset value of each write data and the next adjacent write data in turn; after classifying all offset values according to a normalized calculation method, determining the write amount of the write data of each category, the write amount ratio to the total write amount of all write data, and the write count ratio of the write data of each category to the total write count of all write data; and taking the write amount ratio and write count ratio of each category as the analysis result of the category.
[0014] In combination with the first aspect, in one implementation, the specified order is the writing order of the written data, and the normalization calculation method is a power of 2.
[0015] In combination with the first aspect, in one embodiment, the method further includes the following steps: determining the write amount and write times of different types of write data.
[0016] In combination with the first aspect, in one embodiment, the different types include different sizes and different requests, and the process of determining the write amount and write times of different types of write data includes: after obtaining the write amount and write times of different types of write data, determining the write amount of each type of write data, and the write amount ratio of the total write amount of all write data; and determining the write times of each type of write data, and the write times ratio of the total write times of all write data; and using the determined write amount ratio and write times ratio of each type as the analysis result of this type.
[0017] In combination with the first aspect, in one embodiment, the method further includes the following steps: graphically displaying the analysis results.
[0018] In a second aspect, an embodiment of the present application provides an analysis device for block storage write data, wherein the analysis device for block storage write data includes a processor, a memory, and an analysis program for block storage write data stored on the memory and executable by the processor, wherein when the analysis program for block storage write data is executed by the processor, the steps of the method provided in the first aspect are implemented.
[0019] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored an analysis program for block storage writing data. When the analysis program for block storage writing data is executed, the steps of the method provided in the first aspect are implemented.
[0020] Compared with the prior art, the advantages of the present invention are:
[0021] Through multi-dimensional data analysis, the present invention deeply explores the behavioral patterns of written data, providing strong support for storage system performance optimization; specifically, comprehensive analysis including the time dimension, space dimension, data continuity dimension and data size distribution dimension can fully reveal the complexity of data writing behavior.
[0022] At the same time, the present invention can be applied to block storage systems of different sizes and types to cope with various write scenarios; whether it is small enterprise storage or large data center, the present invention can provide efficient and accurate analysis support.
[0023] On this basis, the benefits that the present invention can bring are:
[0024] Improved performance: By accurately identifying performance bottlenecks and optimizing headroom, this invention helps storage systems achieve improved performance and significantly reduce data write latency. This is crucial for improving service responsiveness and user experience.
[0025] Enhanced stability: Through in-depth analysis of I / O write behavior, this invention can promptly identify and address potential failure points, improving the stability and reliability of the storage system. This helps reduce system failures and maintenance costs, ensuring business continuity.
[0026] Optimize resource utilization: This invention helps users allocate storage resources more rationally, avoid resource waste, and reduce operating costs. Through precise resource allocation and optimization, users can maximize the efficiency of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0028] Figure 1 Schematic diagram of a flow chart of a method for analyzing block storage write data in an embodiment of the present invention;
[0029] Figure 2 This is a schematic diagram of the hardware structure of an analysis device for block storage writing data involved in the embodiment of the present application. DETAILED DESCRIPTION
[0030] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0031] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.
[0032] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0033] In a first aspect, an embodiment of the present application provides a method for analyzing block storage write data, the method comprising the steps of: performing a multi-dimensional analysis on a write data storage log, specifically:
[0034] (1) Time dimension: Divide the written data into fixed periods and count the amount and number of written data in each period. This allows us to determine the data fluctuations in different time periods so that we can perform targeted optimizations later. For example, we can adjust the data writing or storage method during the "peak period" (i.e., the time period with more written data).
[0035] In one embodiment, the process of counting the write amount and write times of write data in each cycle includes: after obtaining the write amount and write times of write data in each cycle, determining the write amount of write data in each cycle and the write amount ratio to the total write amount of all write data; and the write times of write data in each cycle and the write times ratio to the total write times of all write data; using the write amount ratio and write times ratio of each cycle as the time dimension analysis result of the cycle.
[0036] For example, the time axis is divided into time periods of 60 minutes to obtain multiple cycles (cycle_0, cycle_1, ...), and the write amount and write times in each cycle are counted, and their percentages of the total are calculated.
[0037] After the time dimension analysis is completed, you can intuitively view the data fluctuations in different time periods through the write volume ratio and write frequency ratio of each cycle.
[0038] (2) Spatial dimension: The space for storing written data is divided into several storage areas according to the preset storage length (they can be divided according to the same or identical storage length during execution), and the amount of written data and the number of writes in each storage area are counted. In this way, the data density in different storage areas can be obtained, and then the hot data areas and cold data areas can be located for subsequent targeted optimization, such as increasing the storage space in the hot data area and reducing the storage space in the cold data area.
[0039] In one embodiment, the process of counting the amount of write data and the number of write times in each storage area includes: after obtaining the amount of write data and the number of write times in each storage area, determining the amount of write data in each storage area and the ratio of the total amount of write data of all write data; and the ratio of the number of write times of write data in each storage area and the total number of write times of all write data; and using the ratio of the amount of write data and the ratio of the number of write times of each storage area as the spatial dimension analysis result of the storage area.
[0040] For example, a storage volume is divided into multiple blocks (block_0, block_1, ...) according to a fixed length (such as 1M), and the write volume and write times in each block are counted, and their percentages in the total are calculated.
[0041] After the spatial dimension analysis is completed, the data density of different storage areas can be intuitively viewed through the write volume ratio and write frequency ratio of each storage area.
[0042] (3) Data continuity dimension: Determine the offset of adjacent written data, and count the amount of data written and the number of writes at the same offset. In this way, the quality of data continuity can be obtained by the number of times corresponding to the offset size. For example, the weight of the amount of data written and the number of writes with a small offset is high, which means that the data continuity is good.
[0043] In one embodiment, the process of determining the offset of adjacent write data and counting the write amount and write count of write data at the same offset includes: sorting the write data in a specified order, and counting the offset value of each write data and the next adjacent write data in turn; after classifying all offset values according to a normalized calculation method, determining the write amount of the write data of each category, and the write amount ratio to the total write amount of all write data; and the write count ratio of the write data of each category, and the write count ratio to the total write count of all write data; and using the write amount ratio and write count ratio of each category as the data continuity dimension analysis result of the category.
[0044] On this basis, the above-mentioned specified order is the writing order of the written data, and the above-mentioned normalization calculation method is a power of 2 (such as 0, 2, 4, 8, ..., 1024, etc.).
[0045] (4) Data size distribution dimension: Determine the write amount and write frequency of different types of write data. In this way, the size distribution of the write data can be obtained based on the write amount and write frequency of each type, thereby understanding the load characteristics of the system.
[0046] In one embodiment, the above-mentioned different types include different sizes and different requests, and the process of determining the write amount and write frequency of different types of write data includes: after obtaining the write amount and write frequency of different types of write data, determining the write amount of each type of write data and the write amount ratio of the total write amount of all write data; and determining the write frequency ratio of each type of write data and the write frequency ratio of the total write frequency of all write data; the determined write amount ratio and write frequency ratio of each type are used as the data size distribution dimension analysis result of this type.
[0047] After completing the multi-dimensional analysis of the data storage log, the multi-dimensional analysis results are visualized; that is, the dimensional analysis results of each of the above dimensions are graphically displayed. These charts can help users intuitively understand the trends and anomalies of IO behavior, providing data support for performance optimization and troubleshooting.
[0048] See below Figure 1 As shown, the above method is described through a specific embodiment.
[0049] S1: Perform multi-dimensional analysis on the logs written to the data storage. The specific steps include:
[0050] (1) Time dimension: Divide the write data into fixed periods, obtain the write amount and write times of the write data in each period, and then determine the write amount of the write data in each period and its ratio to the total write amount of all write data; as well as the write times of the write data in each period and its ratio to the total write times of all write data; use the write amount ratio and write times ratio of each period as the time dimension analysis results of the period.
[0051] (2) Spatial dimension: The space storing the written data is divided into several storage areas according to the same storage length. After obtaining the written data amount and the number of writes in each storage area, the written data amount of each storage area and the ratio of the written data amount to the total written data amount of all written data are determined; as well as the ratio of the number of writes of the written data in each storage area to the total number of writes of all written data are determined; the ratio of the written data amount and the ratio of the number of writes in each storage area are used as the spatial dimension analysis results of the storage area.
[0052] (3) Data continuity dimension: Count the offset values of each written data and the next adjacent written data in the order of writing; after classifying all offset values according to the power of 2, determine the write amount of each category of written data and its ratio to the total write amount of all written data; and the number of writes of each category of written data and its ratio to the total number of writes of all written data; take the write amount ratio and write number ratio of each category as the data continuity dimension analysis results of that category.
[0053] (4) Data size distribution dimension: The process of determining the write amount and write frequency of different types of write data includes: after obtaining the write amount and write frequency of different types of write data, determining the write amount of each type of write data and the write amount ratio of the total write amount of all write data; and determining the write frequency ratio of each type of write data and the write frequency ratio of the total write frequency of all write data; the determined write amount ratio and write frequency ratio of each type are used as the data size distribution dimension analysis results of that type.
[0054] S2: Visual display: After exporting the dimensional analysis results of each of the above dimensions in CSV format, use programming languages and visualization tools (such as Python's matplotlib library, etc.) to generate line charts, bar charts, heat maps, etc. based on the exported data (original data values and accumulated data values) to intuitively display the multi-dimensional analysis results of the written data.
[0055] In a second aspect, an embodiment of the present application provides an analysis device for block storage write data. The analysis device for block storage write data may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.
[0056] Reference Figure 2 , Figure 2 Schematic diagram of the hardware structure of the block storage write data analysis device involved in the embodiment of the present application. In the embodiment of the present application, the block storage write data analysis device may include a processor, a memory, a communication interface and a communication bus.
[0057] The communication bus may be of any type and is used to interconnect the processor, memory, and communication interface.
[0058] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces, which are used to interconnect components within the analysis device used for block storage and writing data, as well as interfaces used to interconnect the analysis device used for block storage and writing data with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet, fiber, or ATM interfaces; user devices can be displays, keyboards, and other devices.
[0059] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0060] The processor may be a general-purpose processor, which may call an analysis program for block storage write data stored in a memory and execute the analysis method for block storage write data provided in an embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the analysis program for block storage write data is called may refer to the various embodiments of the analysis method for block storage write data of the present application, and will not be repeated here.
[0061] Those skilled in the art will understand that Figure 2The hardware structure shown in the figure does not constitute a limitation to the present application and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0062] In a third aspect, an embodiment of the present application also provides a computer-readable storage medium.
[0063] The computer-readable storage medium of the present application stores an analysis program for block storage write data, wherein when the analysis program for block storage write data is executed by a processor, the steps of the above-mentioned analysis method for block storage write data are implemented.
[0064] Among them, the method implemented when the block storage write data analysis program is executed can refer to the various embodiments of the block storage write data analysis method of this application, and will not be repeated here.
[0065] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of the present application.
[0067] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.
[0068] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.
[0069] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.
[0070] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.
[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of the present application.
[0072] The above are only specific implementations of the embodiments of the present invention, but the scope of protection of the embodiments of the present invention is not limited to them. Any person skilled in the art can easily conceive of various equivalent modifications or replacements within the technical scope disclosed in the embodiments of the present invention, and such modifications or replacements should be included in the scope of protection of the embodiments of the present invention. Therefore, the scope of protection of the embodiments of the present invention should be based on the scope of protection of the claims.
Claims
1. A method for analyzing block storage write data, characterized in that: The method comprises the following steps: Divide the written data into fixed cycles and count the amount and number of written data in each cycle; Divide the space for storing written data into several storage areas according to the preset storage length, and count the amount of written data and the number of write times in each storage area; Determine the offset of adjacent written data and count the amount of written data and the number of write times at the same offset.
2. The method for analyzing block storage write data according to claim 1, wherein: The process of counting the write amount and write times of the write data in each cycle includes: after obtaining the write amount and write times of the write data in each cycle, determining the write amount of the write data in each cycle and the write amount ratio to the total write amount of all write data; and the write times of the write data in each cycle and the write times ratio to the total write times of all write data; and using the write amount ratio and write times ratio of each cycle as the analysis result of the cycle.
3. The method for analyzing block storage write data according to claim 1, wherein: The process of counting the amount of data written and the number of writes in each storage area includes: after obtaining the amount of data written and the number of writes in each storage area, determining the amount of data written in each storage area and the ratio of the amount of data written to the total amount of all data written; and the ratio of the number of writes of the data written in each storage area to the total number of writes of all data written; and using the ratio of the amount of data written and the ratio of the number of writes of each storage area as the analysis result of the storage area.
4. The method for analyzing block storage write data according to claim 1, wherein: The process of determining the offset of adjacent write data and counting the write amount and write count of write data at the same offset includes: sorting the write data in a specified order, and counting the offset value of each write data and the next adjacent write data in turn; after classifying all offset values according to a normalized calculation method, determining the write amount of the write data of each category, the write amount ratio to the total write amount of all write data, and the write count ratio of the write data of each category to the total write count of all write data; and taking the write amount ratio and write count ratio of each category as the analysis result of the category.
5. The method for analyzing block storage write data according to claim 4, wherein: The specified order is the writing order of the written data, and the normalization calculation method is a power of 2.
6. The method for analyzing block storage write data according to any one of claims 1 to 5, characterized in that: The method further includes the following steps: determining the write amount and the write times of different types of write data.
7. The method for analyzing block storage write data according to claim 6, wherein: The different types include different sizes and different requests. The process of determining the write amount and write times of different types of write data includes: after obtaining the write amount and write times of different types of write data, determining the write amount of each type of write data and the write amount ratio of the total write amount of all write data; and determining the write times of each type of write data and the write times ratio of the total write times of all write data; the determined write amount ratio and write times ratio of each type are used as the analysis results of this type.
8. The method for analyzing block storage write data according to claim 7, wherein: The method further comprises the following steps: graphically displaying the analysis results.
9. An analysis device for block storage write data, characterized in that The device for analyzing block storage write data includes a processor, a memory, and an analysis program for block storage write data stored on the memory and executable by the processor, wherein when the analysis program for block storage write data is executed by the processor, the steps of the method for analyzing block storage write data according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an analysis program for block storage write data, wherein when the analysis program for block storage write data is executed, the steps of the analysis method for block storage write data according to any one of claims 1 to 8 are implemented.