Distributed fusion storage system performance grade evaluation method
By conducting detailed testing and grading evaluation of distributed converged storage systems from multiple functional dimensions, the problem of lack of unified specifications in the existing technology is solved, and a comprehensive evaluation of the performance of distributed converged storage systems and hardware optimization are achieved to meet enterprise needs. .
Patent Information
- Application Number
- CN202410948880.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-22
AI Technical Summary
The existing distributed converged storage system performance assessment methods lack unified specifications, and cannot comprehensively evaluate products and technologies, resulting in the inability to accurately reflect the actual performance and functions of the storage system.
A distributed converged storage system performance level assessment method is proposed, which tests from four dimensions: general function, file business function, object business function, and big data business function, and scores are assigned according to weights, and multi-protocol storage pool is created and multi-protocol Indicators such as interoperability, snapshot function, capacity quota, QoS function, NFS and CIFS protocol interfaces, bucket and object basic operations are tested in detail, and finally graded and evaluated according to the test results.
A comprehensive evaluation of the performance of distributed converged storage systems is realized, which can more accurately reflect the functions and performance of the system, meet the actual needs of enterprises, and provide hardware optimization solutions to improve system performance.
Smart Images

Figure CN119003298A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed fusion storage systems, and in particular to a distributed fusion storage system performance level assessment method. Background Art
[0002] The distributed fusion storage system can effectively integrate various storage types and storage organization forms such as distributed files, objects, big data, etc., support multiple storage protocols, meet various storage business needs, and improve data storage efficiency.
[0003] Currently, data center storage system performance evaluation is mostly based on a single indicator or a combination of multiple indicators, which means that data center storage system performance evaluation methods are still in their infancy. The industry does not yet have a unified distributed converged storage test specification for product and technology evaluation.
[0004] Therefore, in view of this, the existing structure and deficiencies are studied and improved, and a performance level assessment method for distributed fusion storage system is proposed. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a method for evaluating the performance level of a distributed fusion storage system, which solves the problems raised in the above-mentioned background technology.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a distributed fusion storage system performance level evaluation method, the distributed fusion storage system performance level evaluation method comprising the following steps:
[0007] S1. Test indicators:
[0008] The performance level of the distributed converged storage system is tested from four dimensions: general functions, file business functions, object business functions, and big data business functions. The weight of general functions is 20%, the weight of file business functions is 20%, the weight of object business functions is 30%, and the weight of data business functions is 30%.
[0009] The test indicators of common functions include multi-protocol storage pool creation test, multi-protocol interoperability test, snapshot function, capacity quota function, and QoS function;
[0010] The test indicators of file service functions include NFS and CIFS protocol interface tests, capacity quota function, file number quota function, and AI domain connection function;
[0011] The test indicators of object business functions include basic operations on buckets and objects, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function;
[0012] The test indicators of big data business functions include HDFS function, Yarn & MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test;
[0013] S2. Test steps for creating a multi-protocol storage pool:
[0014] Enter the storage pool management interface of the storage management system;
[0015] Perform storage creation operations, enter storage pool-related configurations, including at least the storage pool name, pool redundancy policy, and data security level, and submit the creation and view the storage pool creation process;
[0016] After the storage pool is created, check the storage pool information;
[0017] Enable file, object, and big data services on each storage node;
[0018] Verify that stored files, objects, and big data services are normal;
[0019] S3. Test steps for multi-protocol interoperability test:
[0020] Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform;
[0021] The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol;
[0022] View files on different protocol hosts respectively;
[0023] Download all files from different protocol hosts respectively and compare the file MD5;
[0024] S4. Testing steps for snapshot function:
[0025] Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform;
[0026] The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol;
[0027] Log in to the storage system and create a snapshot snaptest1 for the namespace;
[0028] View the snapshot snaptest1 data;
[0029] S5. Test steps for the capacity quota function in the general function:
[0030] Set the capacity quota to 1G through the storage management interface;
[0031] Mount the shared directory on the Windows host and Linux host using CIFS and NFS respectively, and complete the docking configuration between the big data platform and the storage;
[0032] The Windows host writes a 200M file in the CIFS shared directory, the Linux host writes a 200M file in the NFS shared directory, the HDFS host writes a 200M file through the HDFS protocol, and the object host uploads a 200M file through the S3 protocol;
[0033] Test hosts with different protocols write 300M of data to the namespace again;
[0034] S6. Test steps for the QoS function:
[0035] The NFS test host uses the Vdbench test tool to continuously write data to the shared directory;
[0036] After the test host's service runs for 2 minutes, the performance reaches the stable value A;
[0037] Log in to the storage management interface, set the namespace QoS policy, limit the performance to B, where B < A, and observe the business changes;
[0038] After a preset time, turn off the namespace QoS policy and observe the business changes;
[0039] The object test host uses the CosBench test tool to continuously write objects to the storage;
[0040] After the test host's service runs for 2 minutes, the performance reaches the stable value C;
[0041] Log in to the storage management interface, set the namespace QoS policy, limit the performance to D, where D < C, and observe the business changes;
[0042] After a preset time, turn off the namespace QoS policy and observe the business changes;
[0043] The big data test host runs the TestDFSIO component to perform the write performance test;
[0044] After the test host's service runs for 2 minutes, the performance reaches the stable value E;
[0045] Log in to the storage management interface, set the namespace QoS policy, limit the performance to F, where F < E, and observe the changes in the service;
[0046] After a preset time, turn off the namespace QoS policy and observe the changes in the service;
[0047] Test steps for S7, NFS, and CIFS protocol interfaces:
[0048] Create a local directory / mnt / nfstest on the Linux test host and mount the share via the standard NFS protocol;
[0049] Perform operations such as creating, renaming, modifying attributes, and deleting directories in the NFS shared directory;
[0050] Perform operations such as creating, writing, reading, modifying, renaming, modifying attributes, and deleting files in the NFS shared directory;
[0051] On the Windows test host, select "Map Network Drive", choose a drive letter in the drive, and map the share via the standard CIFS protocol;
[0052] Perform operations such as creating, renaming, and deleting directories in the CIFS shared directory;
[0053] Perform operations such as creating, writing, reading, modifying, renaming, and deleting files in the CIFS shared directory;
[0054] View the contents in the shared directory through the Linux test host and the Windows test host respectively;
[0055] Test steps for the capacity quota function in the file service function of S8:
[0056] Set the space quota to 1G through the storage management interface;
[0057] Mount the shared directory on the test host via the standard NFS protocol;
[0058] The test host writes 900MB of data to the shared directory;
[0059] The test host writes another 200MB of data to the shared directory;
[0060] Test steps for the file number quota function of S9:
[0061] Set the file number quota to 1000 through the storage management interface;
[0062] Mount the shared directory on the test host via the standard NFS protocol;
[0063] The test host writes 900 files to the shared directory;
[0064] The test host writes another 200 files to the shared directory;
[0065] S10. Test steps for connecting to AI domain function:
[0066] In the storage management interface, create a namespace testad, configure CIFS sharing, and set aduser to have full control permissions;
[0067] On the Windows test host, select "Map network drive", select the drive letter in the drive, use the AD domain user aduser to map the shared directory testad, and perform file operations;
[0068] S11. Test steps for basic operations on buckets and objects:
[0069] Use the S3 Browser tool to create a bucket named basictest using the test account basic.
[0070] Use S3 Browser to query the bucket information of the account;
[0071] Use S3 Browser to upload, download, and delete objects;
[0072] Delete an empty bucket on the S3 Browser management interface;
[0073] S12. Test steps for bucket ACL permission control:
[0074] Use the S3 Browser tool to create a bucket testbucket using the account bucketacl and upload the object testobject1;
[0075] Use the S3 Browser tool to configure access control for bucket testbucket using account bucketacl1, and authorize bucket reading for account bucketacl2;
[0076] Use the S3 Browser tool to list objects in bucket testbucket using account bucketacl2, and upload object testobject2.
[0077] Use the S3 Browser tool to configure access control for bucket testbucket using account bucketacl1, grant write permission to the bucket to account bucketacl2, and obtain related permissions;
[0078] Use the S3 Browser tool to upload the object testobject2 using the account bucketacl2;
[0079] Use the S3 Browser tool to delete the bucket testbucket using the account bucketacl1;
[0080] S13, bucket policy control test steps:
[0081] Use S3 Browser to add the account's access certificate, create a bucket named policytest using account bucketpolicy1, and upload objects object1 and object2.
[0082] Use the S3 Browser tool to set access policies for other accounts to the bucket, grant bucket enumeration permissions to accounts bucketpolicy2 and bucketpolicy3, and grant account bucketpolicy2 permissions to read and upload resources in the bucket;
[0083] Use the S3 Browser tool to read objects object1 and object2 using account bucketpolicy2.
[0084] Use the S3 Browser tool to read objects object1 and object2 using account bucketpolicy3.
[0085] Use the S3 Browser tool to upload objects object3 and object4 using account bucketpolicy2.
[0086] Use the S3 Browser tool to upload objects object3 and object4 using account bucketpolicy3.
[0087] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy2.
[0088] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy3.
[0089] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy1.
[0090] S14. Test steps for uploading objects in multiple segments:
[0091] The user uses S3 Browser to add the account multipart access certificate and create a bucket;
[0092] The user turns on the multipart upload feature of S3 Browser and uses the tool to upload a large object;
[0093] Download the object using the S3 Browser tool;
[0094] S15. Test steps for bucket log function:
[0095] Use the S3 Browser tool to create a bucket named bucketlogtest using the test account bucketlog.
[0096] Right-click the bucket bucketlogtest, select Bucket Logging Settings, and enable the bucket logging function;
[0097] Perform operations such as upload, download, and delete on the bucket bucketlogtest;
[0098] After waiting for 10 minutes, check whether there is a directory in the bucket that records logs;
[0099] S16. Test steps for HDFS functional test:
[0100] Switch to the hdfs user on the test host, create a folder, and execute the statement: hdfsdfs-mkdir-p / hdfs_test / mydir
[0101] View the folder and execute the statement: hdfs dfs-ls / hdfs_test / ;
[0102] Delete the folder and execute the statement: hdfs dfs-rm-r / hdfs_test / mydir;
[0103] View the folder and execute the statement: hdfs dfs-ls / hdfs_test / ;
[0104] To upload a file, specify a local file. Run the following command: hdfs dfs-put / tmp / testAccess.txt / hdfs_test / , where / tmp / testAccess.txt is a local file. Then run the following command to check whether the file exists: hdfs dfs-ls / hdfs_test / .
[0105] Download the file to the local computer and execute the statement: hdfs dfs-get / hdfs_test / testAccess.txt / tmp / testAccess2.txt;
[0106] Delete the file and execute the statement: hdfs dfs-rm / hdfs_test / testAccess.txt. Then execute the command to check whether the file exists: hdfs dfs-ls / hdfs_test / ;
[0107] S17, Test steps for Yarn & MapReduce functional test:
[0108] Switch to yarn user on the test host and submit the MapReduce teragen job. Refer to the execution statement: yarnjar / opt / cloudera / parcels / CDH6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / hadoop-mapreduce-examples.jar teragen 10000000 / tmp / teragen;
[0109] During the task execution, you can open another client to view the MapReduce job. Refer to the execution statement: mapred job-list;
[0110] The task has been completed. You can view all tasks by executing the following statement: mapred job-list all;
[0111] Submit the terasort job, refer to the execution statement: yarn jar / opt / cloudera / parcels / CDH-6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / ha;doop-mapreduce-examples.jar terasort / tmp / teragen / tmp / terasort;
[0112] Verify that terasort is executed successfully. Refer to the execution statement: yarn jar / opt / cloudera / parcels / CDH-6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / hadoop;
[0113] mapreduce-examples.jar teravalidate / tmp / terasort / tmp / teravalidate;
[0114] S18. Test steps for basic operation test of Hbase table:
[0115] Switch to the hbase user on the test host and execute hbase shell to enter the Hbase CLI view;
[0116] Create a table and execute the statement: create'test','f1','f2','f3'. After completion, execute list to check whether the table exists.
[0117] Disable table, execute statement: disable'test', and then execute describe'test';
[0118] Alter table, execute statement: alter'test',{NAME=>'f3',METHOD=>'delete'}, and then execute describe'test' after completion;
[0119] Enable table, execute statement: enable'test', and then execute describe'test' after completion;
[0120] Drop table: first execute disable'test', then execute drop'test', and finally execute list;
[0121] S19. Test steps for SparkSQL basic operation test:
[0122] Switch to the spark user on the test host and execute spark-shell to enter the spark CLI view;
[0123] Use create statement to create a table and execute the statement: sqlContext.sql("create external table test(a int,b string)row format delimited fields terminated by','stored astextfile").show;
[0124] Execute the statement to check whether the table test exists: sqlContext.sql("show tables").show;
[0125] Alter table, execute statement: sqlContext.sql("alter table test rename to mytest").show;
[0126] Execute the statement to check whether the table name is modified: sqlContext.sql("show tables").show;
[0127] Drop table, execute statement: sqlContext.sql("drop table mytest").show;
[0128] Execute the statement to check whether the table mytest exists: sqlContext.sql("show tables").show;
[0129] S20, Hive basic operation test steps:
[0130] Switch to the hive user on the test host and execute hive to enter the hive CLI view;
[0131] Use the create statement to create a table and execute the statement: create external table test(a int,bstring)row format delimited fields terminated by','stored as textfile location' / tmp / hive_test / test1';
[0132] Execute the statement to check whether the table test exists: show tables;
[0133] Alter table, execute statement: alter table test rename to mytest;
[0134] Execute the statement to check whether the table name is changed to mytest: show tables;
[0135] Drop table, execute statement: drop table mytest;
[0136] Execute the statement to check whether the mytest table exists: show tables.
[0137] Furthermore, in step S1, the three-level indicators of multi-protocol storage pool creation test, multi-protocol intercommunication test, snapshot function, capacity quota function, and QoS function account for 4% each;
[0138] The three-level indicators of NFS and CIFS protocol interface testing, capacity quota function, file number quota function, and AI domain connection function account for 5% each;
[0139] The three-level indicators of basic operations on buckets and objects, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function account for 6% each;
[0140] The third-level indicators of HDFS function test, Yarn & MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test account for 6% each.
[0141] Furthermore, in step S2, the expected result of the test step of the multi-protocol storage pool creation test is: the storage pool is successfully created, each node service is successfully started, and the stored files, objects, and big data services are normal;
[0142] In step S3, the expected results of the test steps of the multi-protocol interoperability test are: CIFS and NFS are mounted successfully, the big data platform and storage are connected, all protocol hosts are operated successfully, files are listed successfully, all protocol hosts see all files, files are downloaded successfully, and the MD5 of files downloaded by all protocol hosts are consistent.
[0143] Furthermore, in step S4, the expected result of the test step of the snapshot function is: CIFS and NFS are mounted successfully, the big data platform and the storage are connected, all protocol host operations are successful, the snapshot is created successfully, and the snapshot data is consistent with the source file content;
[0144] In step S5, the expected results of the test step of the capacity quota function in the general function are: the quota is set successfully, CIFS and NFS are mounted successfully, the big data platform and storage are connected, data is written successfully, and the file fails to be written in excess of the quota;
[0145] In step S6, the expected result of the test step of the QoS function is: the QoS is successfully set for the namespace, the performance meets the QoS setting, and the service performance is restored after the namespace policy is closed.
[0146] Further, in step S7, the expected results of the test steps of the NFS and CIFS protocol interface tests are: successful mounting, successful operation, and consistent viewing content;
[0147] In step S8, the expected results of the test step of the capacity quota function in the file service function are: quota setting success, mounting success, data writing success, and data over-quota writing failure;
[0148] In step S9, the expected results of the test step of the file quota function are: quota setting success, mounting success, data writing success, and data writing failure due to excess of quota;
[0149] In step S10, the expected result of the test step of connecting to the AI domain function is: CIFS share configuration is successful, mapping is successful, and file operation is successful.
[0150] Furthermore, in step S11, the expected results of the test steps of the basic operations of buckets and objects are: successful creation of buckets, successful query of bucket information, successful upload, download and deletion of objects, and successful deletion of buckets;
[0151] In step S12, the expected results of the test step of bucket ACL permission control are: bucketacl1 successfully creates a bucket and uploads an object; sets bucket permissions and obtains bucket permissions successfully; bucketacl2 successfully lists objects and fails to upload objects; sets bucket permissions and obtains bucket permissions successfully; bucketacl2 successfully uploads an object; bucketacl1 successfully deletes a bucket;
[0152] In step S13, the expected results of the test step of bucket policy control are: bucket creation success, object upload success, bucket access policy setting success, object reading success, object reading failure, object upload success, object upload failure, object deletion failure, object deletion failure, object deletion success;
[0153] In step S14, the expected results of the test step of uploading an object in a multi-part manner are: the bucket is successfully created, the user successfully uploads the object using the multi-part upload function, and the object is successfully downloaded;
[0154] In step S15, the expected results of the test step of the bucket log function are: the bucket is created successfully, the bucket log function is enabled successfully, the object operation is successful, and the upload, download, and deletion logs can be viewed.
[0155] Further, in step S16, the expected results of the test step of the HDFS functional test are: successful execution, no error prompt; the query result contains the created mydir directory; successful execution, no error prompt; the query result has the mydir directory; successful execution, no error prompt, and the query command query result shows that there is a testAccess.txt file in the / hdfs_test directory; successful execution, no error prompt, and the local " / tmp" directory has a testAccess2.txt file; successful execution, no error prompt, and the query command query result shows that there is no testAccess.txt file in the " / hdfs_test" directory;
[0156] In step S17, the expected results of the test step of the Yarn & MapReduce functional test are: the mapreduce job is submitted successfully, the execution is successful, the information such as Jobld and State can be viewed, and Terasort is executed successfully;
[0157] In step S18, the test step expected results of the basic operation test of the Hbase table are: hbase shell is executed successfully and enters the Hbase view; create table is executed successfully, and list can be executed to check that the table already exists; execution is successful, and the table status is DISABLED; execution is successful, and the table status column family f3 no longer exists; execution is successful, and the table status is ENABLED; execution is successful, and list can be executed to check that the table no longer exists;
[0158] In step S19, the expected results of the test steps of the SparkSQL basic operation test are: successfully enter the Spark CLI; execute the create table statement without any error information, show tables, and after checking the table, the newly created table test will be displayed; check that the table name has been changed to mytest; table mytest is successfully deleted, and show tables is executed again to find that the table mytest has been deleted;
[0159] In step S20, the expected results of the operation steps of the Hive basic operation test are: successfully entering the hive CLI, executing the table creation statement without any error information, showing the newly created table test, showing the table name has been changed to mytest through showtables, and showing that the table mytest has been deleted successfully, and showing that the table mytest has been deleted through show tables.
[0160] Furthermore, the distributed fusion storage system performance level assessment method further includes the following steps:
[0161] S21. Scoring criteria:
[0162] For each functional test, points are assigned according to the weights shown. In order to promote distributed storage with more innovative functions, object business functions and big data business functions are weighted 30% each, and basic general functions and file business functions are weighted 20% each. If each test is passed, full marks are awarded according to the weight, otherwise, the item is recorded as 0 points;
[0163] According to the scoring results, the distributed fusion storage function test is divided into five levels, L1-L5. The distributed fusion storage solutions of each level can be described as follows:
[0164] L1: Below 20, the solution is difficult to apply to enterprise business and needs to be restructured and the design strengthened;
[0165] L2: greater than or equal to 20, less than 40, the distributed converged storage solution is not fully functional and cannot support the business development of the enterprise;
[0166] L3: greater than or equal to 40 and less than 60. The distributed fusion storage solution is generally designed and basically usable, but there is much room for improvement.
[0167] L4: The score is greater than or equal to 60 and less than 80%. The distributed converged storage solution is well designed and there is still room for improvement.
[0168] L5: The score is greater than or equal to 80 points. The distributed converged storage solution is well designed and can support enterprises to carry out distributed business.
[0169] Furthermore, the distributed fusion storage system performance level assessment method further includes the following steps:
[0170] S22. Information Records:
[0171] When performing the functional test in the above steps S2-S21, the corresponding test results are recorded based on the system hardware, that is, different specifications of hardware are used so that the system will produce corresponding test results, and the hardware name and its selling price in each sales channel are also recorded;
[0172] S23. Hardware optimization:
[0173] The hardware is ranked based on the test results and hardware prices, and hardware optimization solutions are recommended based on the system installed on the current hardware.
[0174] Furthermore, in step S23, the hardware optimization solution can directly display the price required after upgrading the current hardware and the improvement of the system test results after optimization.
[0175] The present invention provides a distributed fusion storage system performance rating method, which has the following beneficial effects:
[0176] 1. This distributed fusion storage system performance rating method tests the main functions of the distributed fusion storage system from four dimensions: general functions, file business functions, object business functions, and big data business functions, and evaluates the test results in a graded manner, comprehensively evaluating whether the design of the data center distributed fusion storage solution meets the application needs of the enterprise. In the process of distributed fusion storage evaluation, specific scoring reference standards are proposed for each indicator, and different score weights are assigned according to the importance of each indicator, which makes the evaluation of the data center distributed fusion storage solution closer to the actual needs of users.
[0177] 2. The distributed fusion storage system performance level assessment method tests various system functions on different hardware to record the effect of different specifications of hardware on system functions, that is, the simpler the hardware, the lower the effect, and the more sophisticated the hardware, the higher the effect. After recording the test results of different hardware on the corresponding test functions in the system, the selling prices under various sales channels are collected based on the hardware names. Therefore, when testing various system functions in the current hardware, other test results that are better than the test results are listed upward based on the test results, and the corresponding hardware and the prices of various hardware channels are displayed based on other test results, and the gap between other test results and the test results is displayed, thereby providing a reference for hardware optimization solutions when optimizing a certain function in the future. BRIEF DESCRIPTION OF THE DRAWINGS
[0178] Figure 1 A schematic diagram of test items of a distributed fusion storage system performance level assessment method of the present invention;
[0179] Figure 2 A test network diagram of a distributed fusion storage system performance level assessment method of the present invention;
[0180] Figure 3 A schematic diagram of a press configuration of a distributed fusion storage system performance level assessment method of the present invention. DETAILED DESCRIPTION
[0181] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0182] like Figure 1-Figure 3 As shown, the present invention provides a technical solution: a distributed fusion storage system performance level evaluation method, the distributed fusion storage system performance level evaluation method comprises the following steps:
[0183] First, the distributed fusion storage test network is as follows Figure 2 As shown, it is mainly composed of press, distributed storage, 10G switch, 1G switch, control terminal, etc.;
[0184] The press configuration is as follows Figure 3 As shown, the test environment temperature can be selected to be 15℃~35℃ and the relative humidity can be 25%~75%;
[0185] S1. Test indicators:
[0186] The performance level of the distributed converged storage system is tested from four dimensions: general functions, file business functions, object business functions, and big data business functions. The weight of general functions is 20%, the weight of file business functions is 20%, the weight of object business functions is 30%, and the weight of data business functions is 30%.
[0187] The test indicators of common functions include multi-protocol storage pool creation test, multi-protocol interoperability test, snapshot function, capacity quota function, and QoS function;
[0188] The test indicators of file service functions include NFS and CIFS protocol interface tests, capacity quota function, file number quota function, and AI domain connection function;
[0189] The test indicators of object business functions include basic operations on buckets and objects, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function;
[0190] The test indicators of big data business functions include HDFS function, Yarn & MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test;
[0191] The third-level indicators of multi-protocol storage pool creation test, multi-protocol intercommunication test, snapshot function, capacity quota function, and QoS function account for 4% each;
[0192] The three-level indicators of NFS and CIFS protocol interface testing, capacity quota function, file number quota function, and AI domain connection function account for 5% each;
[0193] The three-level indicators of basic operations on buckets and objects, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function account for 6% each;
[0194] The third-level indicators of HDFS function test, Yarn & MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test account for 6% each;
[0195] S2. Test steps for creating a multi-protocol storage pool:
[0196] Enter the storage pool management interface of the storage management system;
[0197] Perform storage creation operations, enter storage pool-related configurations, including at least the storage pool name, pool redundancy policy, and data security level, and submit the creation and view the storage pool creation process;
[0198] After the storage pool is created, check the storage pool information;
[0199] Enable file, object, and big data services on each storage node;
[0200] Verify that stored files, objects, and big data services are normal;
[0201] Among them, the expected results of the test steps of the multi-protocol storage pool creation test are: the storage pool is created successfully, the services of each node are started successfully, and the stored files, objects, and big data services are normal;
[0202] S3. Test steps for multi-protocol interoperability test:
[0203] Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform;
[0204] The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol;
[0205] View files on different protocol hosts respectively;
[0206] Download all files from different protocol hosts respectively and compare the file MD5;
[0207] Among them, the expected results of the test steps of the multi-protocol interoperability test are: CIFS and NFS are mounted successfully, the big data platform and storage are connected, all protocol host operations are successful, files are listed successfully, all protocol hosts see all files, files are downloaded successfully, and the MD5 of files downloaded by all protocol hosts are consistent;
[0208] S4. Testing steps for snapshot function:
[0209] Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform;
[0210] The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol;
[0211] Log in to the storage system and create a snapshot snaptest1 for the namespace;
[0212] View snapshot snaptest1 data;
[0213] Expected results of the test steps for the snapshot function: Mounting is successful for CIFS and NFS, the big data platform is successfully docked with the storage, all protocol host operations are successful, snapshot creation is successful, and the snapshot data is consistent with the source file content;
[0214] S5. Test steps for the capacity quota function in the general function:
[0215] Set the capacity quota to 1G through the storage management interface;
[0216] Mount the shared directory on the Windows host and Linux host using CIFS and NFS respectively, and complete the docking configuration between the big data platform and the storage;
[0217] The Windows host writes a 200M file in the CIFS shared directory, the Linux host writes a 200M file in the NFS shared directory, the HDFS host writes a 200M file through the HDFS protocol, and the object host uploads a 200M file through the S3 protocol;
[0218] Test hosts with different protocols write another 300M of data to the namespace;
[0219] Expected results of the test steps for the capacity quota function in the general function: Quota setting is successful, mounting is successful for CIFS and NFS, the big data platform is successfully docked with the storage, data writing is successful, and file overwriting beyond the quota fails;
[0220] S6. Test steps for the QoS function:
[0221] The NFS test host uses the Vdbench test tool to continuously write data to the shared directory;
[0222] After the test host's service runs for 2 minutes, the performance reaches the stable value A;
[0223] Log in to the storage management interface, set the namespace QoS policy to limit the performance to B, where B < A, and observe the changes in the service;
[0224] After a preset time, turn off the namespace QoS policy and observe the changes in the service;
[0225] The object test host uses the CosBench test tool to continuously write objects to the storage;
[0226] After the test host's service runs for 2 minutes, the performance reaches the stable value C;
[0227] Log in to the storage management interface, set the namespace QoS policy to limit the performance to D, where D < C, and observe the changes in the service;
[0228] After the preset time, turn off the namespace QoS policy and observe the changes in the service;
[0229] The big data test host runs the TestDFSIO component to perform the write performance test;
[0230] After the test host service runs for 2 minutes, the performance reaches the stable value E;
[0231] Log in to the storage management interface, set the namespace QoS policy, limit the performance to F (F < E), and observe the changes in the service;
[0232] After the preset time, turn off the namespace QoS policy and observe the changes in the service;
[0233] Expected results of the test steps for the QoS function: The QoS for the namespace is successfully set, the performance conforms to the QoS setting, and the service performance resumes after the namespace policy is turned off;
[0234] Test steps for the S7, NFS, and CIFS protocol interface tests:
[0235] Create a local directory / mnt / nfstest on the Linux test host and mount the share via the standard NFS protocol;
[0236] Perform operations such as creating, renaming, modifying attributes, and deleting directories in the NFS shared directory;
[0237] Perform operations such as creating, writing, reading, modifying, renaming, modifying attributes, and deleting files in the NFS shared directory;
[0238] On the Windows test host, select "Map Network Drive", select a drive letter in the drive, and map the share via the standard CIFS protocol;
[0239] Perform operations such as creating, renaming, and deleting directories in the CIFS shared directory;
[0240] Perform operations such as creating, writing, reading, modifying, renaming, and deleting files in the CIFS shared directory;
[0241] View the contents in the shared directory through the Linux test host and the Windows test host respectively;
[0242] Expected results of the test steps for the NFS and CIFS protocol interface tests: The mount is successful, the operations are successful, and the viewed contents are consistent;
[0243] Test steps for the capacity quota function in the S8 file service function:
[0244] Set the space quota to 1G through the storage management interface;
[0245] Mount the shared directory on the test host using the standard NFS protocol;
[0246] The test host writes 900MB of data to the shared directory;
[0247] The test host writes another 200MB to the shared directory;
[0248] The expected results of the test steps for the capacity quota function in the file business function: quota setting success, mounting success, data writing success, and data writing failure due to excess quota;
[0249] S9. Test steps for the file quota function:
[0250] Set the file quota to 1000 through the storage management interface;
[0251] Mount the shared directory on the test host using the standard NFS protocol;
[0252] The test host writes 900 files to the shared directory;
[0253] The test host writes another 200 files to the shared directory;
[0254] Expected results of the test steps for the file quota function: quota setting is successful, mounting is successful, data writing is successful, and data writing exceeds the quota fails;
[0255] S10. Test steps for connecting to AI domain function:
[0256] In the storage management interface, create a namespace testad, configure CIFS sharing, and set aduser to have full control permissions;
[0257] On the Windows test host, select "Map network drive", select the drive letter in the drive, use the AD domain user aduser to map the shared directory testad, and perform file operations;
[0258] Expected results of the test steps for connecting to the AI domain function: CIFS share configuration is successful, mapping is successful, and file operations are successful;
[0259] S11. Test steps for basic operations on buckets and objects:
[0260] Use the S3 Browser tool to create a bucket named basictest using the test account basic.
[0261] Use S3 Browser to query the bucket information of the account;
[0262] Use S3 Browser to upload, download, and delete objects;
[0263] Delete an empty bucket on the S3 Browser management interface;
[0264] Expected results of the test steps for basic operations on buckets and objects: successful bucket creation, successful bucket information query, successful object upload, download, and deletion, and successful bucket deletion;
[0265] S12. Test steps for bucket ACL permission control:
[0266] Use the S3 Browser tool to create a bucket testbucket using the account bucketacl and upload the object testobject1;
[0267] Use the S3 Browser tool to configure access control for bucket testbucket using account bucketacl1, and authorize bucket reading for account bucketacl2;
[0268] Use the S3 Browser tool to list objects in bucket testbucket using account bucketacl2, and upload object testobject2.
[0269] Use the S3 Browser tool to configure access control for bucket testbucket using account bucketacl1, grant write permission to the bucket to account bucketacl2, and obtain related permissions;
[0270] Use the S3 Browser tool to upload the object testobject2 using the account bucketacl2;
[0271] Use the S3 Browser tool to delete the bucket testbucket using the account bucketacl1;
[0272] Expected results of the test steps for bucket ACL permission control: bucketacl1 successfully creates a bucket and uploads an object; sets bucket permissions and obtains bucket permissions successfully; bucketacl2 successfully lists objects and fails to upload objects; sets bucket permissions and obtains bucket permissions successfully; bucketacl2 successfully uploads an object; bucketacl1 successfully deletes a bucket;
[0273] S13, bucket policy control test steps:
[0274] Use S3 Browser to add the account's access certificate, create a bucket named policytest using account bucketpolicy1, and upload objects object1 and object2.
[0275] Use the S3 Browser tool to set access policies for other accounts to the bucket, grant bucket enumeration permissions to accounts bucketpolicy2 and bucketpolicy3, and grant account bucketpolicy2 permissions to read and upload resources in the bucket;
[0276] Use the S3 Browser tool to read objects object1 and object2 using account bucketpolicy2.
[0277] Use the S3 Browser tool to read objects object1 and object2 using account bucketpolicy3.
[0278] Use the S3 Browser tool to upload objects object3 and object4 using account bucketpolicy2.
[0279] Use the S3 Browser tool to upload objects object3 and object4 using account bucketpolicy3.
[0280] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy2.
[0281] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy3.
[0282] Use the S3 Browser tool to delete objects object1 and object2 using account bucketpolicy1.
[0283] Expected results of the test steps of bucket policy control: bucket creation succeeded, object upload succeeded, bucket access policy setting succeeded, object reading succeeded, object reading failed, object upload succeeded, object upload failed, object deletion failed, object deletion failed, object deletion succeeded;
[0284] S14. Test steps for uploading objects in multiple segments:
[0285] The user uses S3 Browser to add the account multipart access certificate and create a bucket;
[0286] The user turns on the multipart upload feature of S3 Browser and uses the tool to upload a large object;
[0287] Download the object using the S3 Browser tool;
[0288] Expected results of the test steps for uploading objects in multipart mode: bucket creation succeeds, the user successfully uploads an object using the multipart upload function, and the object is successfully downloaded;
[0289] S15. Test steps for bucket log function:
[0290] Use the S3 Browser tool to create a bucket named bucketlogtest using the test account bucketlog.
[0291] Right-click the bucket bucketlogtest, select Bucket Logging Settings, and enable the bucket logging function;
[0292] Perform operations such as upload, download, and delete on the bucket bucketlogtest;
[0293] After waiting for 10 minutes, check whether there is a directory in the bucket that records logs;
[0294] Expected results of the bucket log function test steps: bucket creation is successful, bucket log function is successfully enabled, object operation is successful, and upload, download, and deletion logs can be viewed;
[0295] S16. Test steps for HDFS functional test:
[0296] Switch to the hdfs user on the test host, create a folder, and execute the statement: hdfsdfs-mkdir-p / hdfs_test / mydir
[0297] View the folder and execute the statement: hdfs dfs-ls / hdfs_test / ;
[0298] Delete the folder and execute the statement: hdfs dfs-rm-r / hdfs_test / mydir;
[0299] View the folder and execute the statement: hdfs dfs-ls / hdfs_test / ;
[0300] To upload a file, specify a local file. Run the following command: hdfs dfs-put / tmp / testAccess.txt / hdfs_test / , where / tmp / testAccess.txt is a local file. Then run the following command to check whether the file exists: hdfs dfs-ls / hdfs_test / .
[0301] Download the file to the local computer and execute the statement: hdfs dfs-get / hdfs_test / testAccess.txt / tmp / testAccess2.txt;
[0302] Delete the file and execute the statement: hdfs dfs-rm / hdfs_test / testAccess.txt. Then execute the command to check whether the file exists: hdfs dfs-ls / hdfs_test / ;
[0303] Among them, the expected results of the test steps of the HDFS functional test are: successful execution, no error prompt; the query result includes the created mydir directory; successful execution, no error prompt; the query result includes the mydir directory; successful execution, no error prompt, and the query command query result shows that there is a testAccess.txt file in the / hdfs_test directory; successful execution, no error prompt, and the local " / tmp" directory contains the testAccess2.txt file; successful execution, no error prompt, and the query command query result shows that there is no testAccess.txt file in the " / hdfs_test" directory;
[0304] S17, Test steps for Yarn & MapReduce functional test:
[0305] Switch to yarn user on the test host and submit the MapReduce teragen job. Refer to the execution statement: yarnjar / opt / cloudera / parcels / CDH6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / hadoop-mapreduce-examples.jar teragen 10000000 / tmp / teragen;
[0306] During the task execution, you can open another client to view the MapReduce job. Refer to the execution statement: mapred job-list;
[0307] The task has been completed. You can view all tasks by executing the following statement: mapred job-list all;
[0308] Submit the terasort job, refer to the execution statement: yarn jar / opt / cloudera / parcels / CDH-6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / ha;doop-mapreduce-examples.jar terasort / tmp / teragen / tmp / terasort;
[0309] Verify that terasort is executed successfully. Refer to the execution statement: yarn jar / opt / cloudera / parcels / CDH-6.2.1-1.cdh6.2.1.p0.1580995 / lib / hadoop-mapreduce / hadoop;
[0310] mapreduce-examples.jar teravalidate / tmp / terasort / tmp / teravalidate;
[0311] The expected results of the test steps of the Yarn & MapReduce functional test are as follows: the mapreduce job is submitted successfully, executed successfully, information such as Jobld and State can be viewed, and Terasort is executed successfully;
[0312] S18. Test steps for basic operation test of Hbase table:
[0313] Switch to the hbase user on the test host and execute hbase shell to enter the Hbase CLI view;
[0314] Create a table and execute the statement: create'test','f1','f2','f3'. After completion, execute list to check whether the table exists.
[0315] Disable table, execute statement: disable'test', and then execute describe'test';
[0316] Alter table, execute statement: alter'test',{NAME=>'f3',METHOD=>'delete'}, and then execute describe'test' after completion;
[0317] Enable table, execute statement: enable'test', and then execute describe'test' after completion;
[0318] Drop table: first execute disable'test', then execute drop'test', and finally execute list;
[0319] Among them, the expected results of the test steps of the basic operation test of the Hbase table are as follows: hbase shell is executed successfully and enters the Hbase view; create table is executed successfully, and the table status is DISABLED when executed; the table status column family f3 is no longer present when executed successfully; the table status is ENABLED when executed successfully; the table status is no longer present when executed successfully;
[0320] S19. Test steps for SparkSQL basic operation test:
[0321] Switch to the spark user on the test host and execute spark-shell to enter the spark CLI view;
[0322] Use create statement to create a table and execute the statement: sqlContext.sql("create external table test(a int,b string)row format delimited fields terminated by','stored astextfile").show;
[0323] Execute the statement to check whether the table test exists: sqlContext.sql("show tables").show;
[0324] Alter table, execute statement: sqlContext.sql("alter table test rename to mytest").show;
[0325] Execute the statement to check whether the table name is modified: sqlContext.sql("show tables").show;
[0326] Drop table, execute statement: sqlContext.sql("drop table mytest").show;
[0327] Execute the statement to check whether the table mytest exists: sqlContext.sql("show tables").show;
[0328] The expected results of the SparkSQL basic operation test are as follows: successfully enter the Spark CLI; execute the create table statement without any error message, and after checking the table with show tables, the newly created table test will be displayed; check that the table name has been changed to mytest; table mytest is deleted successfully, and after executing show tables again, it is found that table mytest has been deleted;
[0329] S20, Hive basic operation test steps:
[0330] Switch to the hive user on the test host and execute hive to enter the hive CLI view;
[0331] Use the create statement to create a table and execute the statement: create external table test(a int,bstring)row format delimited fields terminated by','stored as textfile location' / tmp / hive_test / test1';
[0332] Execute the statement to check whether the table test exists: show tables;
[0333] Alter table, execute statement: alter table test rename to mytest;
[0334] Execute the statement to check whether the table name is changed to mytest: show tables;
[0335] Drop table, execute statement: drop table mytest;
[0336] Execute the statement to check whether the table mytest exists: show tables;
[0337] The expected results of the steps of the Hive basic operation test are as follows: successfully enter the Hive CLI, execute the table creation statement without any error message, and check the table using show tables. The newly created table test is displayed. The table name has been changed to mytest. The table mytest is successfully deleted. The table mytest is deleted using show tables.
[0338] The distributed fusion storage system performance level assessment method also includes the following steps:
[0339] S21. Scoring criteria:
[0340] For each functional test, points are assigned according to the weights shown. In order to promote distributed storage with more innovative functions, object business functions and big data business functions are weighted 30% each, and basic general functions and file business functions are weighted 20% each. If each test is passed, full marks are awarded according to the weight, otherwise, the item is recorded as 0 points;
[0341] According to the scoring results, the distributed fusion storage function test is divided into five levels, L1-L5. The distributed fusion storage solutions of each level can be described as follows:
[0342] L1: Below 20, the solution is difficult to apply to enterprise business and needs to be restructured and the design strengthened;
[0343] L2: greater than or equal to 20, less than 40, the distributed converged storage solution is not fully functional and cannot support the business development of the enterprise;
[0344] L3: greater than or equal to 40 and less than 60. The distributed fusion storage solution is generally designed and basically usable, but there is much room for improvement.
[0345] L4: The score is greater than or equal to 60 and less than 80%. The distributed converged storage solution is well designed and there is still room for improvement.
[0346] L5: The score is greater than or equal to 80 points. The distributed converged storage solution is well designed and can support enterprises to carry out distributed business.
[0347] Based on the above description, the present invention tests the main functions of the distributed fusion storage system from four dimensions: general functions, file business functions, object business functions, and big data business functions, and performs graded evaluation on the test results to comprehensively evaluate whether the design of the distributed fusion storage solution of the data center meets the application needs of the enterprise. In the process of distributed fusion storage evaluation, specific scoring reference standards are proposed for each indicator, and different score weights are assigned according to the importance of each indicator, which makes the evaluation of the distributed fusion storage solution of the data center closer to the actual needs of users.
[0348] The distributed fusion storage system performance level assessment method also includes the following steps:
[0349] S22. Information Records:
[0350] When performing the functional test in the above steps S2-S21, the corresponding test results are recorded based on the system hardware, that is, different specifications of hardware are used so that the system will produce corresponding test results, and the hardware name and its selling price in each sales channel are also recorded;
[0351] S23. Hardware optimization:
[0352] Sort the hardware based on the test results and hardware prices, and recommend hardware optimization solutions based on the systems installed on the current hardware. The hardware optimization solutions can directly show the price required to upgrade the current hardware and the improvement of the system test results after optimization.
[0353] By testing various system functions on different hardware, we can record the effects of different specifications of hardware on system functions. That is, the simpler the hardware, the lower the effect, while the more sophisticated the hardware, the higher the effect. After recording the test results of different hardware on the corresponding test functions in the system, we also collect the selling prices under various sales channels based on the hardware names. Therefore, when testing various system functions in the current hardware, other test results that are better than the test results are listed upward based on the test results, and the corresponding hardware and hardware channel prices are displayed based on other test results, and the gap between other test results and the test results is displayed. Therefore, a reference for hardware optimization solutions can be provided when optimizing a certain function in the future.
[0354] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.
Claims
1. A method for evaluating the performance level of a distributed fusion storage system, characterized in that: The distributed fusion storage system performance level assessment method comprises the following steps: S1. Test indicators: The performance level of the distributed converged storage system is tested from four dimensions: general functions, file business functions, object business functions, and big data business functions. The weight of general functions is 20%, the weight of file business functions is 20%, the weight of object business functions is 30%, and the weight of data business functions is 30%. The test indicators of general functions include multi-protocol storage pool creation test, multi-protocol interoperability test, snapshot function, capacity quota function, and QoS function; the test indicators of file business functions include NFS and CIFS protocol interface test, capacity quota function, file number quota function, and AI domain connection function; the test indicators of object business functions include basic operations on buckets and objects, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function; the test indicators of big data business functions include HDFS function, Yarn & MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test; S2. Test steps for creating a multi-protocol storage pool: Enter the storage pool management interface of the storage management system; Perform storage creation operations, enter storage pool-related configurations, including at least the storage pool name, pool redundancy policy, and data security level, and submit the creation and view the storage pool creation process; After the storage pool is created, check the storage pool information; Enable file, object, and big data services on each storage node; Verify that stored files, objects, and big data services are normal; S3. Test steps for multi-protocol interoperability test: Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform; The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol; View files on different protocol hosts respectively; Download all files from different protocol hosts respectively and compare the file MD5; S4. Testing steps for snapshot function: Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform; The Windows host creates files and writes content in the CIFS shared directory, the Linux host creates files and writes content in the NFS shared directory, the HDFS host writes files using the HDFS protocol, and the target host uploads files using the S3 protocol; Log in to the storage system and create a snapshot snaptest1 for the namespace; View snapshot snaptest1 data; S5. Test steps for capacity quota function in general function: Set the capacity quota to 1G through the storage management interface; Mount the shared directories on the Windows host and Linux host using CIFS and NFS respectively, and complete the interconnection configuration with the storage on the big data platform; The Windows host writes a 200M file to the CIFS shared directory, the Linux host writes a 200M file to the NFS shared directory, the HDFS host writes a 200M file using the HDFS protocol, and the target host uploads a 200M file using the S3 protocol. The test host with different protocols writes another 300M of data to the namespace; S6, QoS function test steps: The NFS test host uses the Vdbench test tool to continuously write data to the shared directory; And observe business changes periodically; Test steps for S7, NFS and CIFS protocol interface testing: Create a local directory / mnt / nfstest on the Linux test host and mount the share using the standard NFS protocol; Create, rename, modify attributes, and delete directories in NFS shared directories; Perform operations such as create, write, read, modify, rename, modify attributes, and delete files in NFS shared directories; perform operations such as create, rename, and delete directories in CIFS shared directories; perform operations such as create, write, read, modify, rename, and delete files in CIFS shared directories; Check the contents of the shared directory through the Linux test host and the Windows test host respectively; S8. Test steps for capacity quota function in file business function: Set the space quota to 1G through the storage management interface; the test host writes 900MB of data to the shared directory; the test host writes another 200MB of data to the shared directory; S9. Test steps for the file quota function: Set the file quota to 1000 through the storage management interface; the test host writes 900 files to the shared directory; the test host writes another 200 files to the shared directory; S10. Test steps for connecting to AI domain function: In the storage management interface, create a namespace testad, configure CIFS sharing, and set aduser to have full control permissions; on the Windows test host, select "Map network drive", select a drive letter in the drive, and use the AD domain user aduser to map the shared directory testad to perform file operations; S11. Test steps for basic operations on buckets and objects: Use the S3 Browser tool to create a bucket named basictest using the test account basic. Query the bucket information of an account; upload, download, and delete objects; delete an empty bucket on the management interface; S12. Test steps for bucket ACL permission control: Use the S3 Browser tool to create a bucket testbucket with each account, upload the object testobject1, configure access control for bucket testbucket, authorize account bucketacl2 to read the bucket, list the objects in bucket testbucket, and upload the object testobject2, configure access control for bucket testbucket, authorize account bucketacl2 to write the bucket, obtain relevant permissions, upload the object testobject2, and delete bucket testbucket; S13, bucket policy control test steps: Use S3 Browser to add the account's access certificate, create a bucket named policytest using account bucketpolicy1, and upload objects object1 and object2. Use the S3 Browser tool to set access policies for other accounts to the bucket, grant bucket enumeration permissions to accounts bucketpolicy2 and bucketpolicy3, and grant account bucketpolicy2 permissions to read and upload resources in the bucket; S14. Test steps for uploading objects in multiple segments: The user uses S3 Browser to add the account multipart access certificate and create a bucket; The user turns on the multipart upload feature of S3 Browser and uses the tool to upload a large object; Download the object using the S3 Browser tool; S15. Test steps for bucket log function: Use the S3 Browser tool to upload, download, and delete bucket log test. After waiting for 10 minutes, check whether there is a directory in the bucket that records logs; S16. Test steps for HDFS functional test: Switch to the hdfs user on the test host, create folders, view folders, delete folders, view folders, upload files (that is, specify files that exist locally), download files to the local computer, and execute corresponding statements; S17, Test steps for Yarn & MapReduce functional test: Switch to the yarn user on the test host, submit a MapReduce teragen job, and refer to the execution statement; S18. Test steps for basic operation test of Hbase table: Switch to the hbase user on the test host, execute hbase shell to enter the Hbase CLI view, and execute the statements that can be performed on each table in the CLI view; The tables include create table, disable table, alter table, enable table, and drop table; S19. Test steps for SparkSQL basic operation test: Switch to the spark user on the test host and execute spark-shell to enter the spark CLI view; Use the create statement to create a table, execute the statement and view it; S20, Hive basic operation test steps: Switch to the hive user on the test host and execute hive to enter the hive CLI view and view the information.
2. A distributed fusion storage system performance rating method according to claim 1, characterized in that: In the step S1, the three-level indicators of multi-protocol storage pool creation test, multi-protocol intercommunication test, snapshot function, capacity quota function, and QoS function account for 4% each; the three-level indicators of NFS and CIFS protocol interface test, capacity quota function, file number quota function, and docking AI domain function account for 5% each; the three-level indicators of bucket and object basic operations, bucket ACL permission control, bucket policy control, multi-segment object upload, and bucket log function account for 6% each; the three-level indicators of HDFS function, Yarn&MapReduce function test, Hbase table basic operation test, SparkSQL basic operation test, and Hive table basic operation test account for 6% each.
3. The method for evaluating the performance level of a distributed fusion storage system according to claim 1, characterized in that: In step S2, the expected results of the test step of the multi-protocol storage pool creation test are: the storage pool is created successfully, the services of each node are started successfully, and the stored files, objects, and big data services are normal; in step S3, the expected results of the test step of the multi-protocol interoperability test are: CIFS and NFS are mounted successfully, the big data platform and storage are connected, all protocol hosts are operated successfully, files are listed successfully, all protocol hosts see all files, files are downloaded successfully, and the MD5 of files downloaded by all protocol hosts are consistent.
4. The method for evaluating the performance level of a distributed fusion storage system according to claim 1, characterized in that: In step S4, the expected results of the test steps of the snapshot function are: successful mounting with CIFS and NFS, the big data platform and storage are connected, all protocol host operations are successful, the snapshot is successfully created, and the snapshot data is consistent with the source file content; in step S5, the expected results of the test steps of the capacity quota function in the general function are: successful quota setting, successful mounting with CIFS and NFS, the big data platform and storage are connected, data writing is successful, and file overwriting fails; in step S6, the expected results of the test steps of the QoS function are: successful QoS setting of the namespace, performance meets the QoS settings, and after closing the namespace policy, the business performance is restored.
5. The method for evaluating the performance level of a distributed fusion storage system according to claim 1, characterized in that: In step S7, the expected results of the test steps of the NFS and CIFS protocol interface test are: successful mounting, successful operation, and consistent content viewing; in step S8, the expected results of the test steps of the capacity quota function in the file business function are: successful quota setting, successful mounting, successful data writing, and failure to write data in excess of the quota; in step S9, the expected results of the test steps of the file number quota function are: successful quota setting, successful mounting, successful data writing, and failure to write data in excess of the quota; in step S10, the expected results of the test steps of the AI domain docking function are: successful CIFS share configuration, successful mapping, and successful file operation.
6. A distributed fusion storage system performance rating method according to claim 1, characterized in that: In step S11, the expected results of the test steps for basic operations of buckets and objects are: successful creation of buckets, successful query of bucket information, successful upload, download and deletion of objects, and successful deletion of buckets; in step S12, the expected results of the test steps for bucket ACL permission control are: bucketacl1 successfully creates buckets and successfully uploads objects; sets bucket permissions and successfully obtains bucket permissions; bucketacl2 successfully lists objects and fails to upload objects; sets bucket permissions and successfully obtains bucket permissions; bucketacl2 successfully uploads objects; bucketacl1 successfully deletes buckets; in step S13, bucket The expected results of the policy control test steps: successful bucket creation, successful object upload, successful bucket access policy setting, successful object reading, failed object reading, successful object upload, failed object upload, failed object deletion, failed object deletion, successful object deletion; in step S14, the expected results of the test steps for uploading objects in multi-segment mode: successful bucket creation, successful object upload by user using the multi-segment upload function, successful object download; in step S15, the expected results of the bucket log function test steps: successful bucket creation, successful bucket log function enablement, successful object operation, and the ability to view upload, download, and deletion logs.
7. A distributed fusion storage system performance rating method according to claim 1, characterized in that: In step S16, the expected result of the test step of the HDFS functional test is: successful execution without error prompt; The query result includes the created mydir directory; the execution is successful without any error message; The query result shows that there is a mydir directory; the execution is successful without any error prompts, and the query command query result shows that there is a testAccess.txt file in the / hdfs_test directory; the execution is successful without any error prompts, and the local " / tmp" directory has a testAccess2.txt file; the execution is successful without any error prompts, and the query command query result shows that there is no testAccess.txt file in the " / hdfs_test" directory; in the step S17, the expected results of the test steps of the Yarn&MapReduce functional test: the mapreduce job is submitted successfully, the execution is successful, the Jobld, State and other information can be viewed, and Terasort is executed successfully; in the step S18, the expected results of the test steps of the basic operation test of the Hbase table: hbase The shell is executed successfully and enters the Hbase view; the create table is successful, and the list command can be executed to view that the table already exists; the execution is successful, and the table status is DISABLED; the execution is successful, and the table status column family f3 no longer exists; the execution is successful, and the table status is ENABLED; the execution is successful, and the list command can be executed to view that the table no longer exists; in the step S19, the expected results of the test steps of the SparkSQL basic operation test are: successfully enter the Spark CLI; execute the create table statement without any error message, show tables, and after viewing the table, the newly created table test will be displayed; check that the table name has been changed to mytest; table mytest is deleted successfully, and show tables is executed again to find that table mytest has been deleted; in the step S20, the expected results of the operation steps of the Hive basic operation test are: successfully enter the hive CLI, execute the create table statement without any error message, show tables, and view the table to show the newly created table test, show tables to view that the table name has been changed to mytest, table mytest is deleted successfully, and show tables to view that table mytest has been deleted.
8. The method for evaluating the performance level of a distributed fusion storage system according to claim 1, characterized in that: The distributed fusion storage system performance level assessment method further comprises the following steps: S21. Scoring criteria: For each functional test, points are assigned according to the weights shown. In order to promote distributed storage with more innovative functions, object business functions and big data business functions are weighted 30% each, and basic general functions and file business functions are weighted 20% each. If each test is passed, full marks are awarded according to the weight, otherwise, the item is recorded as 0 points; According to the scoring results, the distributed fusion storage function test is divided into five levels, L1-L5. The distributed fusion storage solutions of each level can be described as follows: L1: Below 20, the solution is difficult to apply to enterprise business and needs to be restructured and the design strengthened; L2: greater than or equal to 20, less than 40, the distributed converged storage solution is not fully functional and cannot support the business development of the enterprise; L3: greater than or equal to 40 and less than 60. The distributed fusion storage solution is generally designed and basically usable, but there is much room for improvement. L4: The score is greater than or equal to 60 and less than 80%. The distributed converged storage solution is well designed and there is still room for improvement. L5: The score is greater than or equal to 80 points. The distributed converged storage solution is well designed and can support enterprises to carry out distributed business.
9. A distributed fusion storage system performance rating method according to claim 1, characterized in that: The distributed fusion storage system performance level assessment method further comprises the following steps: S22. Information Records: When performing the functional test in the above steps S2-S21, the corresponding test results are recorded based on the system hardware, that is, different specifications of hardware are used so that the system will produce corresponding test results, and the hardware name and its selling price in each sales channel are also recorded; S23. Hardware optimization: The hardware is ranked based on the test results and hardware prices, and hardware optimization solutions are recommended based on the system installed on the current hardware.
10. A distributed fusion storage system performance rating method according to claim 9, characterized in that: In step S23, the hardware optimization solution can directly display the price required after the current hardware is upgraded and the improvement of the system test results after optimization.
Citation Information
Cited By
Shared access storage function test system, method, device, medium and product
CN120560979A
Secondary file sharing method and device, electronic equipment and program product
CN120849368A
