A test method suitable for storage cache

Abstract

The present invention relates to a testing method suitable for storage cache. The method includes: dividing business requests into three scenarios according to the amount of data requested by the business; determining corresponding test tasks for different scenarios: the test task for the scenario where all requests hit the cache is a storage performance test, the test task for the scenario where the business and flushing are basically balanced is a data consistency test, and the test task for the cache breakdown scenario is a business continuity and data consistency test; according to the scenario to which the actual business request belongs, the corresponding test task is executed. Compared with the existing storage system testing methods, the method has a thorough scenario analysis and a strong testing pertinence. The specific test contents are clearly defined for different scenarios, and the method can quickly detect fatal problems such as data inconsistency and business interruption in the storage system, reducing the blindness of storage testing and the probability of missed bugs. The method has good universality and is applicable to all storage systems using a hybrid flash architecture.

Description

Technical Field

[0001] The present invention belongs to the technical field of storage system testing methods, and in particular relates to a testing method suitable for storage cache. Background Art

[0002] At present, in order to reduce costs without reducing the business performance of storage systems, almost all storage vendors on the market have chosen a hybrid flash product architecture (SSD+HDD). In this architecture, SSDs are used as data caches for business acceleration, while HDDs are used for persistent data storage. However, due to the difference in read and write performance between SSDs and HDDs, these storage systems often have some performance or business continuity issues under different business loads.

[0003] Among the existing storage cache testing methods, most of them use business simulation tools such as vdbench to perform continuous stress testing to observe whether there will be any problems under long-term business. However, this testing method has the following disadvantages: 1. Lack of pertinence. If business pressure is not effectively applied, some scenarios may be missed; 2. Strong blindness. If the stress testing time is not enough, bugs may be missed. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the existing storage cache testing methods and avoid the omission of performance and business interruption problems to the greatest extent, the present invention proposes a new testing method suitable for storage cache. The method of the present invention will specifically analyze the IO request process and provide corresponding testing methods and test focuses for different business scenarios.

[0005] Figure 1 The figure is a diagram of the business IO flow of the storage system with a hybrid flash architecture. The business request data received by the storage system will be written to the cache layer first, and then the cache data will be flushed back to the persistent storage HDD regularly according to some common flushing algorithms, such as the elevator algorithm and the LRU algorithm.

[0006] By analyzing the above business processes, we found that business requests usually occur in three scenarios:

[0007] 1) When the amount of business request data is small, all business requests are returned upward after reaching the cache layer. This scenario usually occurs shortly after the environment is deployed. In this method, we name it Scenario 1. This scenario is usually used to test the optimal performance of the storage system. Therefore, the test focus of this scenario in this method is set as: storage performance test.

[0008] 2) When the amount of business request data reaches a certain scale, part of the business request hits the cache, and part flows to persistent storage. This is named Scenario 2 in this method. This scenario is the most common scenario for customers. In this method, the test focus of this scenario is set as: data consistency test.

[0009] 3) When the business request data volume is large and urgent, the cache space has a long time to flush the data in the SSD to the HDD because the write performance of the SSD is much stronger than that of the HDD. At the same time, the SSD data cannot be deleted and the space cannot be released. Over time, the SSD space will be full (i.e., cache breakdown occurs). This is named scenario 3 in this method. The test focus of this scenario in this method is set as: business continuity and data consistency testing.

[0010] Specifically, the present invention provides a test method suitable for storage cache, such as Figure 2 As shown, the method comprises the following steps:

[0011] S1. Divide business requests into three scenarios according to the data volume of business requests: all requests hit the cache scenario, business and back flushing basic balance scenario, and cache breakdown scenario;

[0012] S2. Determine the corresponding test tasks for business requests in different scenarios:

[0013] The test task for the request full cache hit scenario is the storage performance test;

[0014] The test task for business and flushing basic balance scenarios is data consistency testing;

[0015] The test tasks for cache breakdown scenarios are business continuity and data consistency testing;

[0016] S3. Execute the corresponding test tasks according to the scenario to which the actual business request belongs.

[0017] Furthermore, the present invention is applicable to the test method for storage cache, step S1, in which the service requests are divided into three scenarios according to the data volume of the service requests, wherein:

[0018] (1) All requests hit the cache: The amount of data requested by the business is small, and all business requests are returned upward after reaching the cache layer. This scenario usually occurs when the environment has just been deployed.

[0019] (2) Business and flushing scenarios with a basic balance: The amount of data requested by business is moderate, some business requests hit the cache, and some flow to persistent storage. This scenario is the most common scenario for customers.

[0020] (3) Cache breakdown scenario: When the amount of data requested by a business is large and urgent, the data in the cache space SSD cannot be flushed back to the HDD in time, causing the SSD space to be full.

[0021] Furthermore, the test task of the request full hit cache scenario described in step S2 of the test method for storage cache of the present invention is a storage performance test, including:

[0022] (1) Storage system optimal IOPS performance test: Use 4K random IO for testing. When testing the optimal write IOPS, run the same FIO business load twice and take the result of the second run. When testing the optimal read IOPS, run the FIO business load only once and take the result.

[0023] (2) Storage system optimal bandwidth performance test: Use 1M sequential IO for testing. Run the write business load twice, and take the result of the second run as the optimal write business bandwidth data. Run the read business load only once, and take the result of the second run as the optimal read business bandwidth data.

[0024] Preferably, the FIO workload described in the test method (1) for storage cache uses: fio-filename= / dev / vdb-ioengine=libaio-randrepeat=1-thread-time_based-ramp_time=10-direct=1-rw=randwrite / randread-bs=4k-size=5G-numjobs=16

[0025] -iodepth=128-group_reporting-runtime=180-name=test;

[0026] The read and write business load described in (2) uses: fio-filename= / dev / vdb-ioengine=libaio-randrepeat=1-thread-time_based-ramp_time=10-direct=1-rw=write / read-bs=1M-size=5G-numjobs=16-iodepth=128-group_reporting-runtime=180-name=test.

[0027] Furthermore, the test task of the business and flushing basic balance scenario described in step S2 of the test method for storage cache of the present invention is a data consistency test, including:

[0028] (1) Server failure test

[0029] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0030] [Step 2] Reboot / poweroff / manually unplug the power cord / power off via IPMI on any server in the storage cluster to check whether there are any data problems with the existing read and write services;

[0031] [Step 3] Restore the failed server, check the existing read and write services again, and then create a new read and write service to check whether it is normal;

[0032] (2) Disk failure test

[0033] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0034] [Step 2] Unplug a cache disk or data disk from any server in the storage cluster and check whether there are any data problems with the existing read and write services;

[0035] [Step 3] Restore the faulty disk, check the existing read and write services again, and then create a new read and write service to check whether it is normal;

[0036] (3) Network fault testing

[0037] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0038] [Step 2] Perform ifdown / unplug the network cable of any server storage network card in the storage cluster to check whether there are data problems in the existing read and write services;

[0039] [Step 3] Restore the faulty network card, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

[0040] Furthermore, the present invention is applicable to the cache breakdown scenario described in step S2 of the test method for storing cache, and the test task is business continuity and data consistency test, and the business in this scenario includes reading and writing of old business data and reading and writing of new business data, wherein:

[0041] Old business data includes: old business data in the cache; old business data completely flushed to the HDD; old business data partially kept in the cache and partially flushed to the HDD;

[0042] New business data bypasses the cache and is directly read and written to the HDD.

[0043] Furthermore, the present invention is applicable to the cache breakdown scenario described in step S2 of the test method for storage cache, and the test tasks are business continuity and data consistency tests, including:

[0044] (I) Business continuity testing:

[0045] [Step 1] Pre-embed a pure write data service, the data volume of which is 95% of the cache disk space. This service is used to construct cache breakdown.

[0046] [Step 2] Pre-embed 5 read and 5 write services, with a data volume of 10% of the cache disk space. This service is used to test the read and write service continuity of old service data;

[0047] [Step 3] After the cache is penetrated, create a new 7-read 3-write service with a data volume of 5% of the cache disk space. This service runs for a long time. This service is used to test the read and write continuity of new service data.

[0048] (II) Data consistency test:

[0049] (1) Server failure test

[0050] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0051] [Step 2] Reboot / poweroff / manually unplug the power cord / power off via IPMI on any server in the storage cluster to check whether there are any data problems with the existing read and write services;

[0052] [Step 3] Restore the failed server, check the existing read and write services again, and then create a new read and write service to check whether it is normal;

[0053] (2) Disk failure test

[0054] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0055] [Step 2] Unplug a cache disk or data disk from any server in the storage cluster and check whether there are any data problems with the existing read and write services;

[0056] [Step 3] Restore the faulty disk, check the existing read and write services again, and then create a new read and write service to check whether it is normal;

[0057] (3) Network fault testing

[0058] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0059] [Step 2] Perform ifdown / unplug the network cable of any server storage network card in the storage cluster to check whether there are data problems in the existing read and write services;

[0060] [Step 3] Restore the faulty network card, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

[0061] On the other hand, the present invention further provides a testing device applicable to storage cache, and when the device is running, the steps of the above-mentioned testing method applicable to storage cache are executed.

[0062] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned test method applicable to storage cache are implemented.

[0063] In summary, compared with the existing storage system testing methods, the test method of the present invention is suitable for storage cache. The scenario analysis is thorough, the test is highly targeted, and specific test contents are clearly defined for different scenarios. It can quickly detect fatal problems such as data inconsistency and business interruption in the storage system, greatly reducing the blindness of storage testing and the probability of missed bugs. At the same time, the method also has good universality, and all storage systems using a hybrid flash architecture are suitable for this test method. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 The figure is a schematic diagram of the storage system business IO flow of the hybrid flash architecture.

[0066] Figure 2 It is a schematic diagram of the implementation process of the method of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and the present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0068] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.

[0069] Embodiment: A test method suitable for storage cache

[0070] In this embodiment, business requests are divided into the following three scenarios according to the data volume:

[0071] 1) Scenario where all requests hit the cache: When the amount of business request data is small, all business requests are returned upward after reaching the cache layer. This scenario usually occurs shortly after the environment is deployed. In this method, we name it scenario 1. This scenario is usually used to test the optimal performance of the storage system. Therefore, the test focus of this scenario in this method is set as: storage performance test.

[0072] 2) Basic balance scenario of business and flushing: When the amount of business request data reaches a certain scale, part of the business request hits the cache, and part flows to persistent storage. This is named scenario 2 in this method. This scenario is the most common scenario for customers. In this method, the test focus of this scenario is set as: data consistency test.

[0073] 3) Cache breakdown scenario: When the business request data volume is large and urgent, the cache space cannot be flushed back to the HDD in time because the write performance of the SSD is much stronger than that of the HDD. At the same time, the SSD data cannot be deleted and the space cannot be released. Over time, the SSD space will be full. This is named scenario 3 in this method. The test focus of this scenario in this method is set as: business continuity and data consistency test.

[0074] For the three scenarios defined above, this method is tested separately according to the following steps:

[0075] Scenario 1: All requests hit the cache

[0076] This scenario usually occurs when the environment has just been deployed and is usually used to test the optimal performance of the storage system.

[0077] [Storage system optimal IOPS performance test method] If you need to test the optimal IOPS of the storage system, you can use 4K random IO for testing. When testing the optimal write IOPS, run the same FIO business load twice and take the result of the latter run. However, this is not necessary to test the optimal read IOPS, and you only need to run the business load once.

[0078] The business load can use: fio-filename = / dev / vdb-ioengine = libaio-randrepeat = 1-thread-time_based-ramp_time = 10-direct = 1-rw = randwrite / randread-bs = 4k-size = 5G-numjobs = 16-iodepth = 128-group_reporting-runtime = 180-name = test

[0079] [Storage system optimal bandwidth performance test method] If you need to test the optimal read and write bandwidth, you can use 1M sequential IO for testing. As above, the write business load is run twice, and the result of the latter is the optimal write business bandwidth data; the read business load is run once, and the result is the optimal read business bandwidth data.

[0080] The business load can use: fio-filename = / dev / vdb-ioengine = libaio-randrepeat = 1-thread-time_based-ramp_time = 10-direct = 1-rw = write / read-bs = 1M-size = 5G-numjobs = 16-iodepth = 128-group_reporting-runtime = 180-name = test

[0081] Scenario 2: Business and back-flush basic balance scenario

[0082] This scenario usually occurs when the storage environment has been used for a period of time, but the business requests are not urgent and the amount of business data is not large. This scenario focuses on testing data consistency issues.

[0083] In this scenario, there are front-end business requests to read and write cache, and at the same time, the data in the cache is flushed back to the back-end HDD. In this process, the data consistency test results are obtained by injecting the following fault tests:

[0084] 1) Server failure test

[0085] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0086] [Step 2] Reboot / poweroff / manually unplug the power cord / power off via IPMI on any server in the storage cluster to check whether there are any data problems with the existing read and write services;

[0087] [Step 3] Restore the failed server, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

[0088] 2) Disk failure test

[0089] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0090] [Step 2] Unplug a cache disk or data disk from any server in the storage cluster and check whether there are any data problems with the existing read and write services;

[0091] [Step 3] Restore the failed disk, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

[0092] 3) Network fault testing

[0093] [Step 1] Pre-embed data reading and writing services and wait for the services to become stable;

[0094] [Step 2] Perform ifdown / unplug the network cable of any server storage network card in the storage cluster to check whether there are data problems in the existing read and write services;

[0095] [Step 3] Restore the faulty network card, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

[0096] Scenario 3: Cache breakdown

[0097] This scenario occurs because the cached data cannot be flushed back in time and the cache space is full, causing the reading and writing of new data to bypass the cache and be performed directly from the HDD. This scenario focuses on testing business continuity and data consistency issues. The data consistency test method is the same as scenario 2.

[0098] In this scenario, there are two types of services: reading and writing old service data and reading and writing new service data. Due to different access popularity, some old service data exists in the cache, some has been completely flushed to the HDD, and some may have been partially flushed back, resulting in part of it remaining in the cache and part of it being flushed back to the HDD. New service data directly bypasses the cache and is directly read and written to the HDD, but the HDD also carries the service traffic flushed from the cache. The HDD performance may be insufficient, which may lead to interruption of new service reading and writing.

[0099] In summary, the business continuity testing method is as follows:

[0100] [Step 1] Pre-embed a pure write data service, the data volume is 95% of the cache disk space, this service is mainly used to construct cache breakdown;

[0101] [Step 2] Pre-embed 5 read and 5 write services, with a data volume of 10% of the cache disk space. This service is used to test the read and write service continuity of old service data;

[0102] [Step 3] After waiting for the cache to be penetrated, create a new 7-read 3-write service with a data volume of 5% of the cache disk space. Run it for a long time. This service is used to test the read and write continuity of new service data.

[0103] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical contents disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A test method suitable for storage cache, characterized in that: The method comprises: S1. Divide business requests into three scenarios based on the amount of data requested: the scenario where all requests hit the cache, the scenario where business and flushing are basically balanced, and the scenario where the cache is penetrated. (1) Scenario where all requests hit the cache: The amount of data requested by the business is small, and all business requests are returned upward after reaching the cache layer; (2) Business and flushing basic balance scenario: The amount of data requested by the business is moderate, part of the business request hits the cache, and part flows to persistent storage; (3) Cache breakdown scenario: The amount of data requested by the business is large, and the data in the cache space SSD cannot be flushed back to the HDD in time, causing the SSD space to be full; S2. Determine the corresponding test tasks for business requests in different scenarios: The test task for the request full cache hit scenario is the storage performance test; The test task for business and flushing basic balance scenarios is data consistency testing; The test tasks for the cache breakdown scenario are business continuity and data consistency testing; the business continuity testing includes: [Step 1] Pre-embed a pure write data service, the data volume of which is 95% of the cache disk space. This service is used to construct a cache breakdown. [Step 2] Pre-embed 5 read and 5 write services, with a data volume of 10% of the cache disk space. This service is used to test the read and write service continuity of old service data; [Step 3] After the cache is penetrated, create a new 7-read 3-write service with a data volume of 5% of the cache disk space. This service runs for a long time. This service is used to test the read and write continuity of new service data. S3. Execute the corresponding test tasks according to the scenarios to which the actual business requests belong.

2. The test method for storage cache according to claim 1, characterized in that: The test task of the request full hit cache scenario described in step S2 is a storage performance test, including: (1) Storage system optimal IOPS performance test: Use 4K random IO for testing. When testing the optimal write IOPS, run the same FIO business load twice and take the result of the latter run. When testing the optimal read IOPS, run the FIO business load only once and take the result. (2) Storage system optimal bandwidth performance test: Use 1M sequential IO for testing. Run the write business load twice, and take the result of the latter run as the optimal write business bandwidth data; run the read business load only once, and take the result of the run as the optimal read business bandwidth data.

3. The test method for storage cache according to claim 2, characterized in that: The FIO workload described in (1) uses: fio -filename= / dev / vdb -ioengine=libaio -randrepeat=1 -thread -time_based -ramp_time=10 -direct=1 -rw=randwrite / randread -bs=4k -size=5G -numjobs=16 -iodepth=128 -group_reporting -runtime=180 -name=test; The read and write workload described in (2) uses: fio -filename= / dev / vdb -ioengine=libaio -randrepeat=1 -thread -time_based -ramp_time=10 -direct=1 -rw=write / read -bs=1M -size=5G -numjobs=16 -iodepth=128 -group_reporting -runtime=180 -name=test.

4. The test method for storage cache according to claim 1, characterized in that: The test task of the business and flushing basic balance scenario described in step S2 is a data consistency test, including: (1) Server failure test [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Reboot / poweroff / manually unplug the power cord / power off via IPMI on any server in the storage cluster to check whether there are any data problems with the existing read and write services; [Step 3] Restore the failed server, check the existing read and write services again, and then create a new read and write service to check whether it is normal; (2) Disk failure test [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Unplug a cache disk or data disk from any server in the storage cluster and check whether there are any data problems with the existing read and write services; [Step 3] Restore the faulty disk, check the existing read and write services again, and then create a new read and write service to check whether it is normal; (3) Network fault testing [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Perform ifdown / unplug the network cable of any server storage network card in the storage cluster to check whether there are data problems in the existing read and write services; [Step 3] Restore the faulty network card, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

5. The test method for storage cache according to claim 1, characterized in that: The test task of the cache breakdown scenario described in step S2 is a business continuity and data consistency test. The business in this scenario includes the reading and writing of old business data and the reading and writing of new business data, wherein: Old business data includes: old business data in the cache; old business data completely flushed back to the HDD; old business data partially remaining in the cache and partially flushed back to the HDD; New business data bypasses the cache and is directly read and written to the HDD.

6. The test method for storage cache according to claim 5, characterized in that: The test tasks of the cache breakdown scenario described in step S2 are business continuity and data consistency testing, and the data consistency testing includes: (1) Server failure test [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Reboot / poweroff / manually unplug the power cord / power off via IPMI on any server in the storage cluster to check whether there are any data problems with the existing read and write services; [Step 3] Restore the failed server, check the existing read and write services again, and then create a new read and write service to check whether it is normal; (2) Disk failure test [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Unplug a cache disk or data disk from any server in the storage cluster and check whether there are any data problems with the existing read and write services; [Step 3] Restore the faulty disk, check the existing read and write services again, and then create a new read and write service to check whether it is normal; (3) Network fault testing [Step 1] Pre-embed data reading and writing services and wait for the services to become stable; [Step 2] Perform ifdown / unplug the network cable of any server storage network card in the storage cluster to check whether there are data problems in the existing read and write services; [Step 3] Restore the faulty network card, check the existing read and write services again, and then create a new read and write service to check whether it is normal.

7. A test device suitable for storage cache, characterized in that: When the device is running, the steps of the testing method applicable to storage cache described in any one of claims 1-6 are executed.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the test method applicable to a storage cache as claimed in any one of claims 1 to 6 are implemented.

Images