HBase table information test statistical method and device, equipment and storage medium

Abstract

The invention relates to an HBase table information test statistical method and device, equipment and a storage medium. The method comprises the following steps: configuring a to-be-tested data table list input by a user; traversing the to-be-tested data table list, and making a sampling plan for each data table in the to-be-tested data table list; the sampling plan comprises the following steps: sampling according to partitions in a data table, and adding sampling items into a sample pool; constructing a test case based on the sample pool; calling a thread to execute the test case to obtain corresponding performance data; and carrying out statistics on the performance data to obtain a performance test result of each data table. Therefore, the sampling data of different partitions in the data table can be considered to obtain a performance test result with finer granularity, so that the performance test result better conforms to an online actual use condition, and a basis is provided for subsequent performance adjustment and optimization.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for testing and statistically analyzing HBase table information. Background Art

[0002] HBase is a highly reliable, high-performance, column-oriented, scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap server. HBase is a master / slave architecture, and each HBase table generally contains multiple partitions (Regions).

[0003] In traditional technology, performance testing of HBase is generally based on pre-partitioning of an empty test table. The written data is automatically generated by a standard test tool according to the set fixed length and is evenly distributed in each partition. The discreteness when writing is also very good. This is very helpful for the cluster to run a better performance data during testing. According to different fixed length settings, the loading performance difference with large upper and lower limits can be measured.

[0004] However, traditional cluster performance testing methods can only be performed based on the entire cluster, with a relatively coarse granularity. It is impossible to provide a specific read and write performance evaluation of each table under production conditions, nor is it possible to obtain a data feature profile for each table. This results in large differences between the data format and data distribution and the actual online situation, and inaccurate test results. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for testing and statistical HBase table information that can obtain performance test results with finer granularity and meet actual online usage in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for testing and counting HBase table information, including:

[0007] Configure the list of data tables to be tested input by the user;

[0008] Traversing the list of data tables to be tested, and formulating a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool;

[0009] Building test cases based on the sample pool;

[0010] The calling thread executes the test case to obtain corresponding performance data;

[0011] The performance data is counted to obtain the performance test results of each data table.

[0012] In one embodiment, configuring the list of to-be-tested data tables input by the user includes:

[0013] Receive a list of data tables that need to be tested, input by a user, and obtain a list of data tables to be tested;

[0014] Configure the number of sampling entries in the list of data tables to be tested, the number of concurrent threads for stress testing, and the test ratio;

[0015] Each data table in the list of data tables to be tested is divided so that each data table contains two or more partitions.

[0016] In one embodiment, traversing the list of data tables to be tested and formulating a sampling plan for each data table in the list of data tables to be tested includes:

[0017] Traverse and read the data tables in the list of data tables to be tested;

[0018] Count the total number of data entries in the read data table and the number of entries contained in each partition;

[0019] Determine the number of samples for each partition according to the sample interval of each partition of the data table and the number of sampled entries of the data table;

[0020] Determine a sampling range according to the number of samples in each partition, and randomly select a starting point in the sampling range;

[0021] The partition is traversed from the starting point, and sample entries are extracted at preset intervals and added to the sample pool.

[0022] In one embodiment, the step of constructing a test case based on the sample pool includes:

[0023] According to the number of concurrent threads in the stress test, a corresponding number of threads are started concurrently;

[0024] Each thread randomly extracts a number of target data from the sample pool according to the test ratio;

[0025] Parsing the target data to obtain a structured input object;

[0026] Adding the structured input object to a thread-local test queue;

[0027] The test queue is used as a test case for a thread.

[0028] In one embodiment, the calling thread executes the test case to obtain corresponding performance data, including:

[0029] Each thread traverses its own test case to perform corresponding read and write tests;

[0030] When collecting each round of testing, the performance data corresponding to the read and write tests includes: the time of this test, whether it is the current minimum value or the current maximum value.

[0031] In one embodiment, the performing statistics on the performance data to obtain the performance test results of each data table includes:

[0032] After the thread traverses and completes the read and write test of the test case, the test result of the thread is counted, and the test result of the thread includes: at least one of the total read and write time of the thread, the test round, the maximum delay, the minimum delay, the average delay, and the total data throughput;

[0033] After all threads have completed the test, the test results of the data table are statistically calculated based on the test results of each thread. The test results of the data table include: at least one of the total maximum delay, the total minimum delay, the average delay, the total execution time, the average execution time, the total throughput, and the average throughput.

[0034] In a second aspect, the present application also provides an HBase table information test statistics device, including:

[0035] Configuration module, used to configure the list of data tables to be tested input by the user;

[0036] A sampling module is used to traverse the list of data tables to be tested and formulate a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table and adding the sampled items to the sample pool;

[0037] A test case construction module, used to construct a test case based on the sample pool;

[0038] A test module is used to call a thread to execute the test case and obtain corresponding performance data;

[0039] The statistical module is used to collect statistics on the performance data to obtain the performance test results of each data table.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] Configure the list of data tables to be tested input by the user;

[0042] Traversing the list of data tables to be tested, and formulating a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool;

[0043] Building test cases based on the sample pool;

[0044] The calling thread executes the test case to obtain corresponding performance data;

[0045] The performance data is counted to obtain the performance test results of each data table.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] Configure the list of data tables to be tested input by the user;

[0048] Traversing the list of data tables to be tested, and formulating a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool;

[0049] Building test cases based on the sample pool;

[0050] The calling thread executes the test case to obtain corresponding performance data;

[0051] The performance data is counted to obtain the performance test results of each data table.

[0052] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0053] Configure the list of data tables to be tested input by the user;

[0054] Traversing the list of data tables to be tested, and formulating a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool;

[0055] Building test cases based on the sample pool;

[0056] The calling thread executes the test case to obtain corresponding performance data;

[0057] The performance data is counted to obtain the performance test results of each data table.

[0058] The above-mentioned HBase table information test statistics method, device, computer equipment, computer-readable storage medium and computer program product configure the list of data tables to be tested input by the user; thereby, performance testing at the table level granularity can be achieved. Traverse the list of data tables to be tested, and formulate a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool; thereby, the sampling data of different partitions in the data table can be considered, so that the subsequent test results can more truly reflect the online usage. Construct a test case based on the sample pool; call a thread to execute the test case to obtain the corresponding performance data; and perform statistics on the performance data to obtain the performance test results of each data table. In this way, performance test results with finer granularity can be obtained, and can meet the actual online usage, providing a basis for subsequent performance tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] FIG1 is an application environment diagram of an HBase table information test statistics method in one embodiment;

[0061] FIG2 is a flow chart of a method for testing and counting HBase table information in one embodiment;

[0062] FIG3 is a flow chart of a method for testing and counting HBase table information in another embodiment;

[0063] FIG4 is a flow chart of a method for testing and counting HBase table information in another embodiment;

[0064] FIG5 is a structural block diagram of an HBase table information test statistics device in one embodiment;

[0065] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] The HBase table information test statistics method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 101 communicates with the server 102 through the network. First, the user inputs the list of data tables to be tested through the terminal 101, and the server 102 configures the list of data tables to be tested input by the user; traverses the list of data tables to be tested, and formulates a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool; building test cases based on the sample pool; calling the thread to execute the test case to obtain the corresponding performance data; statistics on the performance data to obtain the performance test results of each data table. Finally, the server 102 can feed back the final performance test results to the terminal 101. Among them, the terminal 101 can be but not limited to various personal computers, laptops, smart phones, tablet computers, etc. The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0068] In an exemplary embodiment, as shown in FIG. 2 , a method for testing and counting HBase table information is provided. The method is described by taking the application of the method to the server in FIG. 1 as an example, and includes the following steps 201 to 205 .

[0069] Step 201: configure the list of data tables to be tested input by the user.

[0070] In this embodiment, the user can input a list of data tables that need to be performance tested, and configure the number of sampling entries T for each table, the number of concurrent threads X for stress testing, and the test ratio f∈(0,1]. The test ratio refers to the ratio between the number of test entries in the adopted entries and the total number of entries.

[0071] Exemplarily, a list of data tables that need to be performance tested is received from a user, and a list of data tables to be tested is obtained; the number of sampling entries, the number of concurrent threads for stress testing, and the test ratio of the list of data tables to be tested are configured; each data table in the list of data tables to be tested is divided so that each data table contains two or more partitions.

[0072] In this embodiment, a data table is hierarchically defined, that is, the data table is divided into multiple partitions (regions). Since there may be data imbalance in the partitions in production, and the data between them do not overlap with each other, data stratification according to regions can effectively solve the data balance problem.

[0073] Step 202, traverse the list of data tables to be tested, and formulate a sampling plan for each data table in the list of data tables to be tested.

[0074] The sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool.

[0075] In this embodiment, we first count the total number of data entries S in the current table and formulate a sampling plan for each table. We traverse each region and count the number of data entries Ns contained in the current region. We formulate a sampling plan for each table, calculate the number of samples Sr for each region, determine the sample interval, and define it by system sampling. We calculate the sample interval of the current region as .

[0076] in, .

[0077] Exemplarily, the data tables in the list of data tables to be tested are traversed and read; the total number of data entries in the read data tables and the number of entries contained in each partition are counted respectively; the number of samples for each partition is determined according to the sample interval of each partition of the data table and the number of sampled entries of the data table; the sampling range is determined according to the number of samples of each partition, and a starting point is randomly selected in the sampling range; the partitions are traversed from the starting point, and sample entries are extracted at preset intervals and added to the sample pool.

[0078] For example, a starting point is randomly selected in the current region. Randomly select a starting point within the range. Traverse the current region, extract sample entries at a certain interval, and add them to the sample pool.

[0079] Repeat the above steps until all partitions in the data table are sampled to form the final sample pool.

[0080] Step 203: construct a test case based on the sample pool.

[0081] For example, x threads can be started concurrently, and each thread randomly extracts (Np*f) pieces of data from the sample data pool p according to a given test ratio f, parses them into structured input objects, and adds them to the thread's local test queue L, which is the test case of the current thread. Np represents the total number of entries in the data pool P.

[0082] In this embodiment, a corresponding number of threads are started concurrently according to the number of concurrent threads of the stress test; each thread randomly extracts a number of target data from the sample pool according to the test ratio; the target data is parsed to obtain a structured input object; the structured input object is added to a thread-local test queue; and the test queue is used as a test case for the thread.

[0083] Step 204: calling the thread to execute the test case and obtain corresponding performance data.

[0084] Exemplarily, each thread traverses its own test case to perform corresponding read and write tests; when collecting each round of testing, performance data corresponding to the read and write tests are collected, and the performance data includes: the time of this test, whether it is the current minimum value or the current maximum value.

[0085] In this embodiment, each thread traverses its own test queue L and performs corresponding read and write tests. Each round of testing collects corresponding performance data, such as the time (nanoseconds) of this test and whether it is the current minimum or maximum value. After the thread traversal is completed, the work results of the current thread are counted, including the total read and write time of the current thread, the test round, the maximum delay, the minimum delay, the average delay, and the total data throughput.

[0086] Step 205: Perform statistics on the performance data to obtain the performance test results of each data table.

[0087] In this embodiment, after waiting for all threads to complete execution, the total maximum and minimum delays, average delay, total execution time, average execution time, total throughput, average throughput of the current table, etc. can be counted according to the execution results of each thread, and the test results of the data table can be recorded.

[0088] Exemplarily, after the thread traversal completes the read and write test of the test case, the test results of the thread are counted, and the test results of the thread include: at least one of the total read and write time of the thread, test rounds, maximum delay, minimum delay, average delay, and total data throughput; after all threads have completed the test, the test results of the data table are counted according to the test results of each thread, and the test results of the data table include: at least one of the total maximum delay, total minimum delay, average delay, total execution time, average execution time, total throughput, and average throughput.

[0089] This embodiment can be applied to the read and write test of the data table in the HBase cluster. For example, the user can select a list of data tables for performance testing, and then configure the number of sampled items for each data table, the number of concurrent threads for stress testing, and the sampling ratio executed by each thread. In this way, the performance test of the data table can be performed according to the user's selection, and the test results based on the table granularity can be obtained.

[0090] In this embodiment, by using a region data collection quantity measurement method based on importance sampling, the system sampling interval, the range of randomly selected starting points, and randomly selecting a starting point within the sampling range of the current region for sampling, the concurrent test of read and write performance at the table level granularity is implemented. By sharing the test sample pool, the data preparation delay before the test is greatly reduced. The maximum and minimum delays, average delays, total execution time, average execution time, total throughput, average throughput of the current table, etc. obtained through the test can be used to obtain the business performance evaluation results of the data table.

[0091] In the above-mentioned HBase table information test statistics method, the list of data tables to be tested input by the user is configured; thereby, performance testing at the table-level granularity can be achieved. The list of data tables to be tested is traversed, and a sampling plan is formulated for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool; thereby, the sampling data of different partitions in the data table can be considered, so that subsequent test results can more truly reflect the online usage. Construct a test case based on the sample pool; call a thread to execute the test case to obtain the corresponding performance data; perform statistics on the performance data to obtain the performance test results of each data table. In this way, performance test results with finer granularity can be obtained, and can meet the actual online usage, providing a basis for subsequent performance tuning.

[0092] In another exemplary embodiment, as shown in FIG. 3 , a method for testing and counting HBase table information is provided. The method is described by taking the application of the method to the server in FIG. 1 as an example, and includes the following steps 301 to 305 .

[0093] Step 201: configure the list of data tables to be tested input by the user.

[0094] Step 202, traverse the list of data tables to be tested, and formulate a sampling plan for each data table in the list of data tables to be tested.

[0095] The sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool.

[0096] Step 203: construct a test case based on the sample pool.

[0097] Step 204: calling the thread to execute the test case and obtain corresponding performance data.

[0098] Step 205: Perform statistics on the performance data to obtain the performance test results of each data table.

[0099] In this embodiment, for the specific implementation process and technical effects of steps 201 to 205, please refer to the relevant descriptions of steps 201 to 205 in the method embodiment shown in FIG. 2 , which will not be repeated here.

[0100] Step 306: For all tables in the data list, the performance test result of each data table is output and fed back to the user end.

[0101] In this embodiment, the performance test results of all tables in the data list can be summarized and then fed back to the user end, so that the user can intuitively obtain the test performance results of all data tables in the list, which is convenient for providing a basis for subsequent tuning.

[0102] Exemplarily, based on the method embodiments shown in FIG. 2 and FIG. 3 , as shown in FIG. 4 , another HBase table information test statistics method is provided. First, the user inputs a data table list, the number of sampled entries T for each table, the number of concurrent threads for stress testing X, and the test ratio f executed by each thread. Then, it is determined whether there are data tables in the data table list. If so, the total number of data entries S of the current table is counted, and the sampling plan for each table is specified. The current partition (region) is traversed to calculate the number of samples Sr for each partition. The sample interval is determined, and a starting point is randomly selected. A starting point is randomly selected within the range of the current partition. The data sample entries in the current partition are extracted according to the determined interval and added to the sample pool. Construct a test case: X threads are started concurrently, and each thread randomly extracts (Np*f) pieces of data from the sample pool (which can be defined as a sample data pool P) according to a given test ratio f, parses them into get objects (structured input objects), and adds them to the thread's local test queue L (queue L is the test case of the current thread). Each thread traverses its own test queue L and performs corresponding read and write tests with the data stored in L. After each round of testing, the corresponding performance data is collected, including the time of this test (nanoseconds), whether it is the current minimum or maximum value. After the thread traversal is completed, the work results of the current thread are counted, including the total read and write time of the current thread, test rounds, maximum delay, minimum delay, average delay, and data throughput. Wait for all threads to complete execution, and according to the execution results of each thread, count the total maximum delay, total minimum delay, average delay, total execution time, average execution time, total throughput, and average throughput of the current table. Finally, for all data tables in the list, output the performance test results of each data table and return the results to the user.

[0103] The method in this embodiment has the ability to fully automate testing. Without manual intervention, you only need to set the preset conditions to automatically perform the most accurate sampling based on the online table. After the sampling is completed, multi-threaded read and write stress testing can also be automatically carried out to fully test the real read and write performance of the table in the production environment. Different from the traditional test method for the overall performance of the cluster, the granularity of the test is refined to different tables, thereby meeting the needs of customers for targeted tuning of online applications.

[0104] In addition, the method in this embodiment is more consistent with the actual test results of production. The test of the production table uses an importance sampling algorithm. Unlike simple random sampling, the sampling method of this embodiment takes into account the data volume distribution of different partitions in the table and the weights of different table row sizes. The results of stratified sampling can better reflect the online usage of the production table, so the test results are more consistent with the production process.

[0105] This embodiment can flexibly adapt to different application scenarios, and the method and results of analyzing the structure of the table can also be used as input for the health diagnosis tool of the table, thereby providing a basis for subsequent performance tuning.

[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides an HBase table information test statistics device for implementing the HBase table information test statistics method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more HBase table information test statistics device embodiments provided below can refer to the limitations of the HBase table information test statistics method above, and will not be repeated here.

[0108] In an exemplary embodiment, as shown in FIG5 , a HBase table information test statistics device is provided, including: a configuration module 501, a sampling module 502, a test case construction module 503, a test module 504 and a statistics module 505, wherein:

[0109] Configuration module 501, used to configure the list of data tables to be tested input by the user;

[0110] The sampling module 502 is used to traverse the list of data tables to be tested and formulate a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table and adding the sampled items to the sample pool;

[0111] A test case construction module 503, used to construct a test case based on the sample pool;

[0112] A test module 504 is used to call a thread to execute the test case and obtain corresponding performance data;

[0113] The statistics module 505 is used to collect statistics on the performance data to obtain the performance test results of each data table.

[0114] Exemplarily, the configuration module 501 is specifically used to: receive a list of data tables that need to be performance tested input by a user, and obtain a list of data tables to be tested; configure the number of sampling entries, the number of concurrent threads for stress testing, and the test ratio of the list of data tables to be tested; and divide each data table in the list of data tables to be tested so that each data table contains two or more partitions.

[0115] Exemplarily, the sampling module 502 is specifically used to: traverse and read the data tables in the list of data tables to be tested; respectively count the total number of data entries in the read data tables and the number of entries contained in each partition; determine the number of samples for each partition based on the sample interval of each partition of the data table and the number of sampled entries in the data table; determine the sampling range based on the number of samples for each partition, and randomly select a starting point in the sampling range; traverse the partitions from the starting point, and extract sample entries at preset intervals and add them to the sample pool.

[0116] Exemplarily, the test case construction module 503 is specifically used to: start a corresponding number of threads concurrently according to the number of concurrent threads of the stress test; each thread randomly extracts a number of target data from the sample pool according to the test ratio; parses the target data to obtain a structured input object; adds the structured input object to the thread's local test queue; and uses the test queue as a test case for the thread.

[0117] Exemplarily, the test module 504 is specifically used for: each thread traverses its own test case to perform corresponding read and write tests; when collecting each round of testing, the performance data corresponding to the read and write tests includes: the time of this test, whether it is the current minimum value or the current maximum value.

[0118] Exemplarily, the statistics module 505 is specifically used to: after the thread traversal completes the read and write test of the test case, count the test results of the thread, and the test results of the thread include: at least one of the total read and write time of the thread, test rounds, maximum delay, minimum delay, average delay, and total data throughput; after all threads have completed the test, count the test results of the data table based on the test results of each thread, and the test results of the data table include: at least one of the total maximum delay, total minimum delay, average delay, total execution time, average execution time, total throughput, and average throughput.

[0119] Exemplarily, the above device may further include: a sending module 506, wherein the sending module 506 is configured to output a performance test result of each data table for all tables in the data list, and feed the result back to the user end.

[0120] Each module in the above HBase table information test statistics device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0121] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for testing and statistically analyzing HBase table information is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0122] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0127] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for testing and counting HBase table information, characterized in that: The method comprises: Configure the list of data tables to be tested input by the user; Traversing the list of data tables to be tested, and formulating a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table, and adding the sampled items to the sample pool; Building test cases based on the sample pool; The calling thread executes the test case to obtain corresponding performance data; The performance data is counted to obtain the performance test results of each data table.

2. The method according to claim 1, characterized in that The configuration of the list of data tables to be tested input by the user includes: Receive a list of data tables that need to be tested, input by a user, and obtain a list of data tables to be tested; Configure the number of sampling entries in the list of data tables to be tested, the number of concurrent threads for stress testing, and the test ratio; Each data table in the list of data tables to be tested is divided so that each data table contains two or more partitions.

3. The method according to claim 1, characterized in that The traversing the list of data tables to be tested and formulating a sampling plan for each data table in the list of data tables to be tested includes: Traverse and read the data tables in the list of data tables to be tested; Count the total number of data entries in the read data table and the number of entries contained in each partition; Determine the number of samples for each partition according to the sample interval of each partition of the data table and the number of sampled entries of the data table; Determine a sampling range according to the number of samples in each partition, and randomly select a starting point in the sampling range; The partition is traversed from the starting point, and sample entries are extracted at preset intervals and added to the sample pool.

4. The method according to claim 2, characterized in that: The constructing a test case based on the sample pool includes: According to the number of concurrent threads in the stress test, a corresponding number of threads are started concurrently; Each thread randomly extracts a number of target data from the sample pool according to the test ratio; Parsing the target data to obtain a structured input object; Adding the structured input object to a thread-local test queue; The test queue is used as a test case for a thread.

5. The method according to any one of claims 1 to 4, characterized in that: The calling thread executes the test case to obtain corresponding performance data, including: Each thread traverses its own test case to perform corresponding read and write tests; When collecting each round of testing, the performance data corresponding to the read and write tests includes: the time of this test, whether it is the current minimum value or the current maximum value.

6. The method according to claim 5, characterized in that The performance data is statistically analyzed to obtain the performance test results of each data table, including: After the thread traverses and completes the read and write test of the test case, the test result of the thread is counted, and the test result of the thread includes: at least one of the total read and write time of the thread, the test round, the maximum delay, the minimum delay, the average delay, and the total data throughput; After all threads have completed the test, the test results of the data table are statistically calculated based on the test results of each thread. The test results of the data table include: at least one of the total maximum delay, the total minimum delay, the average delay, the total execution time, the average execution time, the total throughput, and the average throughput.

7. A HBase table information test statistics device, characterized in that: The device comprises: Configuration module, used to configure the list of data tables to be tested input by the user; A sampling module is used to traverse the list of data tables to be tested and formulate a sampling plan for each data table in the list of data tables to be tested; the sampling plan includes: sampling according to the partitions in the data table and adding the sampled items to the sample pool; A test case construction module, used to construct a test case based on the sample pool; A test module is used to call a thread to execute the test case and obtain corresponding performance data; The statistical module is used to collect statistics on the performance data to obtain the performance test results of each data table.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images