An elastic storage-based I / O mode analysis method and device and a storage medium

Abstract

The application discloses an I / O mode analysis method and device based on elastic storage and a storage medium, and the method comprises the following steps: capturing kernel I / O mode in a storage backend; analyzing and abstracting the I / O mode to obtain an analysis result; and importing the captured I / O mode and the analysis result into a new storage scene for reconstruction and playback. By implementing the application, the I / O mode is intercepted from the bottom layer, and the problem that the existing tracking technology cannot detect non-file services is solved. Meanwhile, the analysis method proposes a complete systematic design of extraction-analysis-reinjection, combines the current popular pattern matching and big data analysis and prediction technology, and makes the traditional technology play a greater role in the emerging cloud computing scene. In addition, through the analysis method, the cloud computing front-end service mode can be better understood in the distributed storage scene, and the bottom layer can be more targetedly optimized, so that the end-to-end quality is truly improved.

Description

Technical Field

[0001] This invention relates to the fields of distributed storage, operating system kernel, storage I / O stack and pattern matching prediction technology, and specifically to an I / O pattern analysis method, device and storage medium based on elastic storage. Background Technology

[0002] Since the advent of computers, computing, storage, and networking have been the three main directions of their application development. Among them, storage systems, as the carriers of information, bear the heavy responsibility of data persistence and providing high-performance input / output. Unlike the byte-based access mode of computing and networking, storage devices have their own inherent characteristics, such as a minimum access unit of 512 / 4k bytes, performance varying with I / O mode, and the write amplification and limited media lifetime characteristics brought about by the flash memory storage media that have emerged in recent years.

[0003] To improve the user-friendliness of storage systems, computer software provides various interfaces such as abstractions for files, objects, and blocks, as well as various scheduling strategies and software extensions superimposed in kernel / user space to improve storage performance and efficiency. While this suite of software solutions effectively improves storage reliability and availability, it also significantly increases the complexity of the software architecture and the entire system. For example... Figure 1 The diagram illustrates the call relationships at each level of a standard file I / O process from the application to the hardware storage medium in a single-machine scenario.

[0004] With the development of distributed technologies and the rise of cloud computing services, storage continues to evolve towards high availability, high elasticity, and high performance. However, this also means that the front-end business model based on cloud architecture no longer corresponds one-to-one with the underlying data I / O model. In distributed storage architectures represented by elastic storage, the I / O patterns initiated by clients in the cloud can be vastly different from those of the physical storage system. For example... Figure 2 The diagram shows the conversion between the cloud client business model and the underlying physical node.

[0005] This significant difference means that various optimization solutions designed from the client's perspective for different business models may no longer be effective at the underlying level. Furthermore, when sporadic problems occur in this large and complex system, it is difficult to reliably reproduce and analyze the problem within a very small scope. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method, apparatus and storage medium for I / O pattern analysis based on elastic storage, in order to solve the technical problem that various optimization schemes designed from the client for different business models are no longer effective at the underlying level in the prior art.

[0007] The technical solution proposed in this invention is as follows:

[0008] The first aspect of this invention provides an I / O pattern analysis method based on elastic storage, comprising: capturing kernel I / O patterns at the storage backend; analyzing and abstracting the I / O patterns to obtain analysis results; and importing the captured I / O patterns and analysis results into a new storage scenario for reconstruction and playback.

[0009] Optionally, kernel I / O patterns can be captured at the storage backend, including: capturing kernel I / O patterns at the storage backend based on kernel tracing techniques and tools; or capturing kernel I / O patterns at the storage backend based on kernel modules.

[0010] Optionally, the kernel I / O mode is captured based on the kernel module in the storage backend, including: listening to preset events contained in the kernel block device itself; capturing the preset events when I / O is sent to the underlying hardware; and recording the operations of listening events and capturing events to obtain the captured kernel I / O mode.

[0011] Optionally, the kernel I / O mode is captured based on the kernel module in the storage backend, including: adding a preset function to the I / O transmission of the kernel underlying driver to the forwarding mechanism; and forwarding the captured kernel I / O mode when the kernel underlying driver receives an I / O request.

[0012] Optionally, the I / O patterns are analyzed and abstracted to obtain analysis results, including: extracting and predicting I / O pattern features for multiple access patterns of the I / O patterns based on the Markov prediction model to obtain analysis results, wherein the multiple access patterns include spatial patterns, repetitive behaviors, request sizes, time intervals, and I / O operation types.

[0013] Optionally, I / O mode features are extracted and predicted for multiple access modes of I / O modes based on the Markov prediction model, including prediction based on greedy strategy algorithm, path prediction algorithm or amortization prediction algorithm.

[0014] Optionally, the captured I / O patterns and analysis results are imported into a new storage scenario for reconstruction and playback, including: importing the captured I / O patterns and analysis results into the kernel layer of the new storage scenario; performing adaptation and optimization based on the captured I / O patterns and analysis results to obtain a new I / O pattern; and injecting the new I / O pattern into the underlying hardware to simulate the service scenario.

[0015] A second aspect of the present invention provides an I / O pattern analysis device based on elastic storage, comprising: a capture module for capturing kernel I / O patterns at the storage backend; an analysis module for analyzing and abstracting the I / O patterns to obtain analysis results; and a reconstruction and playback module for importing the captured I / O patterns and analysis results into a new storage scenario for reconstruction and playback.

[0016] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to perform the I / O pattern analysis method based on elastic storage as described in the first aspect and any one of the first aspects of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the I / O pattern analysis method based on elastic storage as described in the first aspect and any one of the first aspects of the present invention.

[0018] The technical solution provided by this invention has the following effects:

[0019] The I / O pattern analysis method, apparatus, and storage medium based on elastic storage provided in this invention intercept I / O patterns from the lowest level, solving the problem that existing tracing technologies are limited by operating platforms and usage scenarios. For example, the inotify tool for Linux file systems is only suitable for tracing metadata modifications of certain file system types and cannot detect non-file business. Furthermore, current mainstream tracing technologies are only used for locating and analyzing problems, without analyzing, summarizing, or deeply extracting the collected I / O models. This analysis method proposes a complete systematic design of extraction-analysis-re-injection, combining current popular pattern matching and big data analysis and prediction technologies to enable traditional technologies to play a greater role in emerging cloud computing scenarios. In addition, through this analysis method, a deeper understanding of cloud computing front-end business models and more targeted optimization of the underlying layers can be achieved in distributed storage scenarios, truly achieving end-to-end quality improvement. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a diagram of a single-machine file I / O call stack;

[0022] Figure 2 This is a schematic diagram of the I / O flow of an elastic storage architecture;

[0023] Figure 3 This is a flowchart of an I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a kernel I / O tracing tool based on the elastic storage-based I / O pattern analysis method according to an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of data flow monitoring based on the I / O pattern analysis method of elastic storage according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of read and write operations of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the blkparse command analysis results of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of the observation results of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of foreground capture features of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the observation results of a scenario-based I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram of the observation results of Scenario 2 of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0032] Figure 12 This is a schematic diagram of the observation results of Scenario 3 of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0033] Figure 13 This is a schematic diagram of the state transition of the I / O pattern analysis method based on elastic storage according to an embodiment of the present invention;

[0034] Figure 14 This is a flowchart of an I / O pattern analysis method based on elastic storage according to another embodiment of the present invention;

[0035] Figure 15 This is a structural block diagram of an I / O pattern analysis device based on elastic storage according to an embodiment of the present invention;

[0036] Figure 16 This is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0037] Figure 17This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] As mentioned in the background section, in a distributed storage architecture represented by China Telecom Cloud Elastic Storage, the I / O patterns of services initiated by cloud clients can differ drastically from those of the physical storage system. This significant difference means that various optimization schemes designed from the client's perspective for different business models may no longer be effective at the underlying level. Furthermore, when sporadic problems occur in this large and complex system, it is extremely difficult to reliably reproduce and analyze the problem within a very small scope.

[0042] To address the significant differences between front-end and back-end operations in single-machine / distributed scenarios, this invention proposes an I / O recording, analysis, and replay design based on existing operating system kernel tracing technology and kernel storage stack architecture. On one hand, it captures I / O patterns at different stages of the entire data transmission path to monitor and analyze the matching relationship between front-end and back-end I / O patterns. On the other hand, it replays the collected data at different times and in different scenarios to simulate and restore scenarios, facilitating business analysis and problem localization.

[0043] At the same time, by combining I / O pattern matching and optimization algorithms with big data technology and accumulating a large amount of detection data, business behavior characteristics can be deduced, and new characteristics can be added to the existing I / O patterns to provide strategies and suggestions for front-end and back-end business optimization.

[0044] According to embodiments of the present invention, an I / O pattern analysis method based on elastic storage is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] This embodiment provides an I / O pattern analysis method based on elastic storage, which can be used in electronic devices such as computers, mobile phones, and tablets. Figure 3 This is a flowchart of an I / O pattern analysis method based on elastic storage according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0046] Step S101: Capture kernel I / O mode at the storage backend.

[0047] In one implementation, when this analysis method is applied to the Linux open-source operating system, built-in kernel monitoring and tracing tools such as blktrace, iostat, or bcc can be used to capture kernel I / O patterns. Among them, blktrace has relatively low-level monitoring points and is equipped with the corresponding data analysis tool blkparse. In addition, the storage stress testing tool fio also supports importing its recorded results as I / O input patterns.

[0048] Specifically, such as Figure 4 As shown, when using kernel tracing techniques and tools to capture kernel I / O patterns, blktrace is used to capture kernel I / O, blkparse is used to analyze the capture results and output pattern records, and fio is used to import pattern records to reproduce the business scenario.

[0049] Furthermore, when kernel tracing techniques and tools are used to capture kernel I / O patterns, their effectiveness can be verified in a single-machine scenario. During verification, the basic command `dd` is used in the foreground user-space client to generate I / O, while `blktrace` is used in the background to capture I / O patterns from kernel space to verify their effectiveness. Specifically, the following steps a-d are used:

[0050] a. First, start blktrace to monitor the data flow of storage device sdb1. The specific method is as follows: Figure 5 As shown.

[0051] b. Simultaneously, use dd to simulate user behavior and initiate read and write operations on the file system located on disk sdb1, such as... Figure 6 As shown.

[0052] c. Use the `blkparse` command to analyze the output of `blktrace`. It combines multiple CPU files generated by `blktrace` and produces human-readable formatted output, such as... Figure 7 As shown.

[0053] d. Import the output obtained from blkparse into fio, and observe again using steps a-c to obtain the following results. Figure 8 The observation results are shown.

[0054] The observation results show that the data transmission model generated by dd fits the I / O model of the subsequent fio simulation, which is in line with the design expectations.

[0055] Meanwhile, kernel tracing techniques and tools can be used to capture kernel I / O patterns, allowing observation of I / O model patterns across various distributed storage scenarios. Given the diverse storage usage scenarios in cloud computing (block storage, file storage, object storage, etc.) and the various distributed storage architectures (replication, erasure coding), kernel tracing techniques are used to observe their different I / O patterns. To facilitate comparison between the front-end and back-end, sequential read / write operations on cloud storage are performed using fio in the front-end. The I / O characteristics captured in the front-end are as follows: Figure 9 As shown.

[0056] Specifically, we can observe its patterns in three scenarios. Scenario 1 involves direct read / write access to the cloud disk from the front end, combined with a three-replica strategy for distributed storage on the back end. The I / O access pattern within the backend distributed storage nodes is as follows: Figure 10 As shown, it can be seen that without intervention from the local file system within the cloud host, the front-end and back-end I / O continuity is similar, but their sizes differ.

[0057] Scenario 2 involves direct front-end read / write access to the cloud disk combined with a back-end distributed storage erasure coding strategy. For example... Figure 11 As shown, in this scenario, the backend I / O is inconsistent with the frontend due to the storage strategy.

[0058] Scenario 3 involves front-end read / write operations on a database based on a distributed file system, combined with a back-end distributed storage system with three replicas. For example... Figure 12 As shown, its I / O characteristics are significantly different from the simple sequential read / write in Scenario 1.

[0059] As can be seen from the above, when capturing kernel I / O patterns on the storage backend using kernel tracing techniques and tools, blktrace can quickly establish an implementation model to verify the design concept. However, it also has certain limitations, including its monitoring points being located above the kernel block device layer, making it unable to completely escape the block layer scheduling mechanism to obtain scenarios where I / O is actually written to disk. Furthermore, blkparse can only perform simple text import and export, lacking pattern analysis and matching capabilities, as well as the ability to communicate with and export data from the database server. Overall, solutions based on existing tools are also limited by cross-software and hardware platform compatibility. To overcome this limitation, kernel I / O patterns can also be captured on the storage backend based on kernel modules.

[0060] In one embodiment, capturing kernel I / O patterns based on kernel modules includes: listening to preset events contained in the kernel block device itself; capturing the preset events when I / O is sent to the underlying hardware; and recording the operations of the listened events and captured events to obtain the captured kernel I / O patterns. Specifically, the `block_rq_issue` event contained in the kernel block device module itself can be listened to, this event can be captured when I / O is sent to the underlying hardware, and the captured kernel I / O patterns obtained from this recorded operation can be forwarded to the user-space client for processing.

[0061] In one embodiment, capturing kernel I / O modes based on kernel modules includes: adding a preset function for forwarding mechanisms to the I / O sending function of the kernel-level driver; and forwarding the captured kernel I / O mode when the kernel-level driver receives an I / O request. Specifically, when capturing kernel I / O modes in this way, a forwarding mechanism is directly added to the I / O sending function scsi_host_template->queuecommand() of the kernel-level driver, so that the driver forwards the I / O request directly to the user-space client for processing as soon as it receives the request.

[0062] Specifically, capturing kernel I / O mode using the above method can synchronously and promptly capture I / O behavior, and is closest to the underlying hardware behavior, eliminating scheduling interference from the block layer and driver internals. However, due to differences in different platforms and underlying hardware, code needs to be injected into different drivers for specific scenarios, which will affect its general applicability to some extent.

[0063] Therefore, based on the above, this analysis method employs two approaches to capture I / O patterns. The first approach involves recording I / O states using operating system kernel tracing techniques and related tools, exporting the I / O recordings to the observation / analysis node, and then using appropriate kernel I / O injection techniques and tools to reproduce the recordings in other scenarios. The second approach involves developing a custom kernel module to record and export I / O at the lowest level of the software stack, and then developing a corresponding suite in user space for I / O replay.

[0064] Step S102: Analyze and abstract the I / O patterns to obtain analysis results. Specifically, based on the Markov prediction model, extract and predict I / O pattern features for multiple access patterns of the I / O patterns to obtain analysis results. The multiple access patterns include spatial patterns, repetitive behaviors, request sizes, time intervals, and I / O operation types.

[0065] Specifically, this analysis method not only records the original I / O patterns and replays and reproduces them in different scenarios, but its greater significance lies in extracting I / O pattern features based on a large amount of business data and statistical records through emerging technologies such as big data, and intelligently and proactively optimizing future business operations. Markov ModelPrediction can be used to extract I / O pattern features and make predictions.

[0066] Theoretically, there are countless possible I / O access patterns. These can be categorized from different dimensions, including spatial patterns, repetitive behavior, request size, time interval, and I / O operation type. Based on the Markov model, this paper analyzes the spatial patterns of I / O requests and derives a corresponding prefetching algorithm. This prefetching algorithm can accurately predict spatial access patterns to a certain extent, thereby reducing I / O response time.

[0067] In the Markov prediction model, Markov is defined as a state where the generation of the next state depends only on the current state, i.e.:

[0068] P[S t+1 |S t ] = P[S t+1 |S1,...,S t ]

[0069] Intuitively, the generation of the next state depends on all historical states, as shown on the right side of the equation. However, Markov's definition ignores historical information and only retains information about the current state to predict the next state. For a specific state S and its next state S', their state transition probability (the probability of transitioning from S to S') is defined as:

[0070] Pss′ =P[S t+1 =s′|S t =s]

[0071] In other words, the generation of the next state is only affected by the current state. If there are a total of n states to choose from, then the state transition matrix P is defined as:

[0072]

[0073] The i-th row of the matrix represents the probabilities that if the current state is i, the next state is 1, ..., n, respectively: Pi1, ..., Pin, and the sum of all probabilities in this row is 1. Therefore, combined with... Figure 13 The state transition matrix can be represented as:

[0074]

[0075] Figure 13 In the diagram, Class1, Class2, Class3, Facebook, Sleep, pub, and Pass represent different states, and there is a transition probability between states.

[0076] Based on the Markov prediction model, several strategies, including greedy algorithms, path prediction algorithms, and amortized prediction algorithms, can be used to predict future business operations. The greedy strategy selects the I / O sequence most likely to migrate from state S based on historical statistics; the more times a single step occurs, the higher the probability. A lower threshold is also given; if the probability is below this threshold, the migration sequence is considered invalid. The path prediction strategy selects the sequence with the highest probability of overall migration paths to provide better prediction accuracy. This strategy uses the Markov model to perform a finite-depth search from the current node to find the path with the highest probability. Amortized prediction: For I / O models that frequently access the same storage location, simply relying on the weights of subsequent steps cannot measure its probability. In this case, it is necessary to consider the frequency of access to that storage location throughout the overall lifecycle of the business operation and apply a weighted approach.

[0077] Specifically, the analysis process based on the Markov prediction model can be implemented in the user-space client. The user-space client also includes a database for unified storage and management of captured I / O patterns. Furthermore, the analysis results can be sent to the user interface for display and interaction.

[0078] Step S103: Import the captured I / O patterns and analysis results into the new storage scenario for reconstruction and playback. Specifically, after importing the captured I / O patterns and analysis results into the new storage scenario, the user-space client of the new storage scenario receives the captured I / O patterns and analysis results and sends them to the kernel layer of the new storage scenario; the kernel module performs adaptation and optimization based on the captured I / O patterns and analysis results to obtain a new I / O pattern; the new I / O pattern is then injected into the underlying hardware to simulate the service scenario. Specifically, during adaptation and optimization, targeted tuning of the I / O services of the underlying storage can be performed, such as adjusting relevant configuration parameters of the storage system and switching the kernel I / O scheduler (noop, cfq, deadline), etc.

[0079] The I / O pattern analysis method based on elastic storage provided in this invention intercepts I / O patterns from the lowest level, solving the problem that existing tracing technologies are limited by operating platforms and usage scenarios. For example, the inotify tool for Linux file systems is only suitable for tracing metadata modifications of certain file system types and cannot detect non-file business operations. Furthermore, current mainstream tracing technologies are only used for locating and analyzing problems, without analyzing, summarizing, or deeply extracting the collected I / O models. This analysis method proposes a complete systematic design of extraction-analysis-re-injection, combining currently popular pattern matching and big data analysis and prediction technologies to enable traditional technologies to play a greater role in emerging cloud computing scenarios. In addition, this analysis method allows for a deeper understanding of cloud computing front-end business models in distributed storage scenarios and more targeted optimization of the underlying layers, truly achieving end-to-end quality improvement.

[0080] In one implementation, such as Figure 14 As shown, the I / O pattern analysis method based on elastic storage adopts the following process: the storage backend captures kernel I / O patterns; I / O patterns are sent in batches to user-space clients for processing; the user-space client performs preliminary analysis and exports I / O pattern files; I / O patterns are incorporated into the database engine for unified management; a large number of I / O patterns are handed over to the pattern analysis engine for big data analysis and abstraction; the analysis and statistical results are sent to the user interface for display and interaction; the original I / O patterns are directly exported to the new storage scenario for reconstruction and playback; the data engine analysis results are superimposed with user input and exported to the new storage scenario for reconstruction and playback; the new storage scenario client receives I / O patterns and sends them to the kernel layer; the kernel module adapts and optimizes storage strategies such as the I / O scheduler according to the pattern characteristics; the kernel module directly injects the new I / O patterns into the underlying hardware to simulate the business scenario.

[0081] The I / O pattern analysis method based on elastic storage provided in this invention is applicable to the comparative analysis and simulation of front-end and back-end I / O business patterns in single-machine and distributed architectures. Its basic design concept is not limited to storage services themselves; computing networks and memory, among other system resources, exhibit high complexity and inconsistency in virtualized and distributed scenarios. This invention is equally applicable to similar problems. The two solutions—based on existing kernel tracing techniques and developing kernel modules—have reference examples in computing network scenarios, while the abstract prediction algorithm based on the Markov model is also applicable to the statistics and prediction of computing resources and network traffic.

[0082] This invention also provides an I / O pattern analysis device based on elastic storage, such as... Figure 15 As shown, the device includes:

[0083] The capture module is used to capture kernel I / O modes at the storage backend; for details, please refer to the corresponding section of the above method embodiment, which will not be repeated here.

[0084] The analysis module is used to analyze and abstract the I / O mode to obtain the analysis results; for details, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0085] The reconstruction and playback module is used to import the captured I / O patterns and analysis results into a new storage environment for reconstruction and playback. For details, please refer to the corresponding sections of the above method embodiments; they will not be repeated here.

[0086] The I / O pattern analysis device based on elastic storage provided in this invention intercepts I / O patterns from the lowest level, solving the problem that existing tracing technologies are limited by operating platforms and usage scenarios. For example, the inotify tool for Linux file systems is only suitable for tracing metadata modifications of certain file system types and cannot detect non-file business. Furthermore, current mainstream tracing technologies are only used for locating and analyzing problems, without analyzing, summarizing, or deeply extracting the collected I / O models. This analysis device proposes a complete systematic design of extraction-analysis-re-injection, combining currently popular pattern matching and big data analysis and prediction technologies to enable traditional technologies to perform better in emerging cloud computing scenarios. In addition, through this analysis device, a deeper understanding of cloud computing front-end business models can be gained in distributed storage scenarios, and more targeted optimizations can be made at the underlying level, truly achieving end-to-end quality improvement.

[0087] For a detailed description of the function of the I / O pattern analysis device based on elastic storage provided in the embodiments of the present invention, please refer to the description of the I / O pattern analysis method based on elastic storage in the above embodiments.

[0088] This invention also provides a storage medium, such as... Figure 16As shown, a computer program 601 is stored on it. When executed by a processor, this program implements the steps of the I / O pattern analysis method based on elastic storage in the above embodiments. The storage medium also stores audio and video stream data, feature frame data, interactive request signaling, encrypted data, and preset data sizes. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0090] This invention also provides an electronic device, such as... Figure 17 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 17 Taking the example of a connection between China and Israel via a bus.

[0091] Processor 51 can be a central processing unit (CPU). Processor 51 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0092] The memory 52, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the corresponding program instructions / modules in the embodiments of the present invention. The processor 51 executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory 52, thereby implementing the I / O pattern analysis method based on flexible storage in the above method embodiments.

[0093] The memory 52 may include a program storage area and a data storage area. The program storage area may store applications required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. Furthermore, the memory 52 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 52 may optionally include memory remotely located relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] The one or more modules are stored in the memory 52, and when executed by the processor 51, they perform the following: Figure 3 -14 shows the I / O pattern analysis method based on elastic storage in the embodiment.

[0095] For specific details regarding the aforementioned electronic devices, please refer to the relevant documentation. Figures 3 to 14 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0096] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for analyzing I / O patterns based on elastic storage, characterized in that, include: Capture kernel I / O patterns at the storage backend; The I / O patterns are analyzed and abstracted to obtain the analysis results; Import the captured I / O patterns and analysis results into the new storage scenario for reconstruction and playback; The analysis and abstraction of the I / O mode yields the following results: Based on the Markov prediction model, I / O mode features are extracted and predicted for multiple access patterns of I / O mode, and analysis results are obtained. The multiple access patterns include spatial patterns, repetitive behavior, request size, time interval and I / O operation type. Based on the Markov prediction model, I / O mode feature extraction and prediction are performed on multiple access modes of I / O patterns, including: Prediction is based on greedy algorithm, path prediction algorithm, or amortization prediction algorithm.

2. The I / O pattern analysis method based on elastic storage according to claim 1, characterized in that, Capture kernel I / O modes on the storage backend, including: Kernel I / O patterns are captured on the storage backend using kernel tracing techniques and tools; or... Kernel I / O patterns are captured based on kernel modules in the storage backend.

3. The I / O pattern analysis method based on elastic storage according to claim 2, characterized in that, The storage backend captures kernel I / O modes based on kernel modules, including: Listen for preset events contained within the kernel block device itself; The preset event is captured when I / O is sent to the underlying hardware; The captured kernel I / O mode is obtained by recording the operations of listening for and capturing events.

4. The I / O pattern analysis method based on elastic storage according to claim 2, characterized in that, The storage backend captures kernel I / O modes based on kernel modules, including: A preset function for adding a forwarding mechanism to I / O transmissions in the kernel-level driver; When the underlying kernel driver receives an I / O request, it forwards the captured kernel I / O pattern.

5. The I / O pattern analysis method based on elastic storage according to claim 1, characterized in that, The captured I / O patterns and analysis results are imported into the new storage scenario for reconstruction and playback, including: Import the captured I / O patterns and analysis results into the kernel layer of the new storage scenario; Based on the captured I / O patterns and analysis results, an adaptation and optimization are performed to obtain a new I / O pattern; The new I / O mode is injected into the underlying hardware to simulate business scenarios.

6. An I / O pattern analysis device based on elastic storage, characterized in that, include: The capture module is used to capture kernel I / O modes at the storage backend; The analysis module is used to analyze and abstract the I / O patterns to obtain analysis results; The reconstruction and playback module is used to import the captured I / O patterns and analysis results into a new storage scenario for reconstruction and playback; The analysis and abstraction of the I / O mode yields the following results: Based on the Markov prediction model, I / O mode features are extracted and predicted for multiple access patterns of I / O mode, and analysis results are obtained. The multiple access patterns include spatial patterns, repetitive behavior, request size, time interval and I / O operation type. Based on the Markov prediction model, I / O mode feature extraction and prediction are performed on multiple access modes of I / O patterns, including: Prediction is based on greedy algorithm, path prediction algorithm, or amortization prediction algorithm.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the I / O pattern analysis method based on elastic storage as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the I / O pattern analysis method based on elastic storage as described in any one of claims 1-5.

Images