Data table performance detection method, system, computing device, and computer-readable storage medium

Abstract

Embodiments of the present specification provide a data table performance detection method, system, computing device and computer readable storage medium, wherein the data table performance detection method comprises: obtaining data operation information of a to-be-detected data table; obtaining data reading time consumption of the to-be-detected data table in a data operation process and time consumption of the data operation process by using the data operation information of the to-be-detected data table; and determining an operation performance index of the to-be-detected data table based on the data reading time consumption and the time consumption of the data operation process.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method for testing the performance of a data table. Background Technology

[0002] With the development of information technology, database data is constantly growing, placing higher demands on database performance. However, during the database development phase, the focus is usually on functionality, with less attention paid to operational performance. As databases become more widely used, quickly manipulating data from massive datasets becomes increasingly difficult. Therefore, it is now common practice to optimize data tables after the database is deployed to improve its operational performance.

[0003] However, given the large number of tables in a database, determining which tables to optimize to improve database performance is a challenge. Therefore, there is an urgent need for a data table performance testing method to help accurately identify the data tables that negatively impact database performance. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for testing the performance of a data table. One or more embodiments of this specification also relate to a data table performance testing system, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data table performance testing method is provided, comprising: acquiring data operation information of a data table to be tested; using the data operation information of the data table to be tested, acquiring the data reading time and the data operation process time of the data table to be tested during the data operation process; and determining the operation performance index of the data table to be tested based on the data reading time and the data operation process time.

[0006] According to a second aspect of the embodiments of this specification, a data table performance testing system is provided, comprising: a testing node, a database service node, a computing node, and multiple storage nodes. The testing node is configured to test the operational performance indicators of a data table in a database using the data table performance testing method as described in any embodiment of this specification; the database service node is configured to run the database and issue data operation tasks for the database to the computing node; the computing node is configured to perform data operation tasks on the data tables of the database; and the storage nodes are configured to store data shards of the data table, wherein the data shards are obtained by splitting the data of the data table using specified fields, and the multiple data shards of the data table are distributed and stored across multiple storage nodes.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the above-described data table performance detection method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data table performance detection method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data table performance detection method.

[0010] One embodiment of this specification implements a data table performance testing method. This method acquires data operation information of the data table to be tested, and uses this information to obtain the data read time and the overall data operation time of the data table during data operations. Based on these data read and operation times, it determines the operational performance indicators of the data table. Therefore, this method determines the impact of data reading on data operations based on the data read time and the overall data operation time, thus obtaining the operational performance indicators of the data table to be tested. These performance indicators can help accurately identify data tables affecting database operation performance, allowing for one or more actions such as optimization or issuing performance alerts, thereby improving the overall operational performance of the database. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of a data table performance testing method provided in one embodiment of this specification;

[0012] Figure 2 This is a flowchart of a data table performance testing method provided in one embodiment of this specification;

[0013] Figure 3 This is a schematic diagram of the implementation architecture of a data table performance testing method provided in one embodiment of this specification in a query application scenario;

[0014] Figure 4 This is a flowchart of the data tilt calculation process provided in one embodiment of this specification;

[0015] Figure 5 This is a flowchart illustrating the calculation and processing procedure for operational performance indicators provided in one embodiment of this specification.

[0016] Figure 6 This is a schematic diagram of a three-round screening process provided in one embodiment of this specification;

[0017] Figure 7 This is a schematic diagram of the structure of a data table performance testing device provided in one embodiment of this specification;

[0018] Figure 8 This is a schematic diagram of the structure of a data table performance testing system provided in one embodiment of this specification;

[0019] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0020] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0021] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0022] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0023] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0024] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0025] Distributed database: A system in which multiple nodes perform data querying, data writing, and other operations in parallel.

[0026] Data skew: In some databases, hash calculations are performed based on the fields specified by the user when creating the table to distribute the data across multiple storage nodes. Therefore, the data in a table can be distributed across multiple storage nodes. If the user-specified fields themselves have uneven data distribution, then during hash calculations and data distribution, some storage nodes may store more data than others, resulting in some nodes storing less or no data at all. This uneven distribution of data is called data skew.

[0027] Data modeling refers to the way tables are created in a database. For example, table creation methods can include settings such as the type of fields, the number of fields, partitioning fields, and sorting fields. It can also include syntax specific to the characteristics of distributed databases, such as specifying fields used for hash calculations.

[0028] Data sharding: When storing data in a database, the data in a table can be divided into multiple data shards using rules such as hash calculations, and then written to multiple storage nodes. Each storage node can have at least one data shard. If a storage node has only one data shard, it reads data using a single thread; if it has more than one, it can read data in parallel using multiple threads.

[0029] In some databases, during data writing, the data in a table is split according to the values ​​of the fields specified by the user when creating the table, distributing the data evenly across multiple storage nodes. This results in a more even distribution of disk space usage across the nodes. Simultaneously, during database operations, data can be read from multiple storage nodes in parallel, improving performance. However, if the specified fields are not evenly distributed during table creation, some storage nodes may exceed their disk capacity limits while others are empty. This leads to significant differences in the amount of data read concurrently across different storage nodes during data operations. Some nodes may have no data to read, while others experience longer read times, impacting overall database performance. Therefore, users need to optimize their databases, but with numerous tables, they often don't know where to begin.

[0030] In view of this, this specification provides a data table performance testing method, and also relates to a data table performance testing device, a data table performance testing system, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0031] See Figure 1 , Figure 1 This diagram illustrates an application scenario of a data table performance testing method according to an embodiment of this specification. Figure 1 As shown, this application scenario may include: a detection node, a database service node, a log repository, a computing node, and multiple storage nodes. The detection node is configured to use the data table performance testing method provided in this embodiment to detect the operational performance indicators of data tables in the database. Specifically, the detection node uses the database's operation logs to obtain data operation information of the data table to be tested, and uses this information to obtain the data read time and the total time spent on the data operation process. Based on these data read and operation times, the operational performance indicators of the data table to be tested are determined. The database service node is configured to run the database and issue data operation tasks to the computing nodes. The computing nodes are configured to execute data operation tasks on the database's data tables. The storage nodes are configured to store data shards of the data tables. These shards are obtained by splitting the data in the data table using specified fields, and the multiple data shards are distributed across multiple storage nodes. The log repository is configured to store database operation logs.

[0032] It should be noted that the database operation logs can be stored in an independent log library, or in the aforementioned storage nodes, database service nodes, or other storage devices; this specification does not impose any restrictions on this.

[0033] This method determines the impact of data reading on data operations based on the data reading time and the overall data operation time, thereby obtaining the operational performance index of the data table to be tested. Based on this operational performance index, it can help people accurately identify the data tables that affect the database operation performance, optimize the data tables or provide performance alerts, and improve the overall operational performance of the database.

[0034] The aforementioned nodes (such as detection nodes, database service nodes, computing nodes, and storage nodes) can be physical servers or cloud servers providing various services. For example, they can be servers providing communication services, data processing services, computing services, storage services, and database services to multiple clients. It should be noted that nodes can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Servers can also be servers in a distributed system, or servers integrated with blockchain. Servers can also be cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0035] It is understood that the above application scenarios are only used to illustrate the methods provided in the embodiments of this specification and do not constitute a limitation on the methods provided in the embodiments of this specification. The methods provided in the embodiments of this specification can perform performance testing on data tables of various databases. For example, the methods provided in the embodiments of this specification can be used for performance testing of data tables in distributed databases, specifically such as the performance testing of data tables in cloud-native data warehouses, to meet the needs of data warehouse optimization.

[0036] See Figure 2 , Figure 2 A flowchart of a data table performance testing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0037] Step 202: Obtain data operation information for the data table to be tested.

[0038] A data table is the basic unit for organizing data in a database. The data table to be tested can be any data table in any type of database. For example, in one or more embodiments, database operation logs can be obtained, and the data tables related to the data operation information recorded in the database operation logs can be used as the data tables to be tested. As another example, in one or more embodiments, a preset filtering strategy can be set according to the needs of the scenario to filter out a portion of the data tables from the database as the data tables to be tested. Yet another example, in one or more embodiments, a detection request can be received, carrying information such as the identifier of the data table to be tested, and the data table to be tested can be determined based on the detection request.

[0039] The data operation information refers to information related to operations performed on the data table to be tested, such as queries. For example, when the data operation task of the data table to be tested is a single-threaded task, the data operation information may include the table identifier (such as the table name), the data operation task identifier, the data read volume of the data operation task, the data read time of the data operation task, and the total time of the entire data operation process of the data operation task. As another example, when the data operation task of the data table to be tested includes a multi-threaded task, the data operation information may include the table identifier of the data table to be tested, the data operation task identifier, multiple subtasks of the data operation task, the data read volume of each subtask, the data read time of each subtask, and the total time of the entire data operation process of the data operation task.

[0040] Step 204: Using the data operation information of the data table to be tested, obtain the data reading time and the data operation time of the data table to be tested during the data operation process.

[0041] The data operation process refers to the process from the issuance of the data operation task to the completion of the data operation task, which includes two stages: data reading and performing operations on the read data.

[0042] It should be noted that the specific method for obtaining the data reading time of the data table under test during the data operation process is related to the data reading method, and the method for obtaining the data reading time needs to be determined according to the data reading method.

[0043] For example, in one or more embodiments, data in a data table is read by a single thread. In this case, the data reading time refers to the time it takes for the thread to read the data.

[0044] For example, in one or more embodiments, the data in a data table is sliced ​​into multiple data shards and stored in multiple storage nodes. When reading, multiple threads read in parallel. In this case, the data reading time refers to the time taken for the thread with the longest data reading time among the multiple parallel threads to read the data.

[0045] Step 206: Based on the data reading time and the data operation process time, determine the operational performance index of the data table to be tested.

[0046] For example, in one or more embodiments, data in a data table is read by a single thread. In this case, the data reading time can be the time it takes for the thread to read the data, such as 2ns, combined with the overall data operation time of the thread, such as 3ns. Then, the operation performance index can be 2 / 3.

[0047] For example, in one or more embodiments, the data of a data table is sliced ​​into multiple data shards and stored in multiple storage nodes. When reading, multiple threads read in parallel. In this case, the data reading time can be the time taken for the thread with the longest data reading time among the multiple parallel threads to read the data, such as 2ns. Combined with the overall data operation time of the thread, such as 4ns, the operation performance index can be 2 / 4.

[0048] By analyzing the time spent reading data and the overall time spent on data operations, we can determine the impact of data reading on data operations, thereby obtaining the operational performance indicators of the data table under test. Therefore, based on these operational performance indicators, we can accurately identify the data tables that affect the database's operational performance, and perform one or more actions such as optimizing the data tables or issuing performance alerts, thereby improving the overall operational performance of the database.

[0049] In one or more embodiments of this specification, for databases, the operational performance of data tables is affected to some extent by data skew, and embodiments for calculating the skewness of data tables are also provided. Specifically, the method further includes:

[0050] The data operation tasks of the data table to be tested are determined using the data operation information of the data table to be tested.

[0051] The data read volume corresponding to multiple subtasks in the data operation task is obtained. The data table to be detected is divided into multiple data shards. The multiple data shards are stored in multiple storage nodes. One of the subtasks is used to read the data of one of the data shards from one of the storage nodes and perform data operations.

[0052] Based on the data reading volume corresponding to the multiple sub-tasks, the data skewness of the data table to be detected is calculated.

[0053] The data skewness is an indicator used to represent the unevenness of data distribution in a data table. Since a subtask reads data from one data shard from a storage node, the unevenness of data distribution in a data table can be calculated based on the data read volumes corresponding to multiple subtasks of a data operation task. The representation of data skewness is not limited, as long as it can represent the degree of unevenness. For example, data skewness can be represented by the ratio of data read volumes corresponding to each subtask. If the ratio of data read volumes corresponding to the four subtasks of a data table is 2:2:4:8, then the data skewness can be represented as 2:2:4:8. As another example, data skewness can be represented by the ratio of the data read volume of the subtask with the largest data read volume to the average of the data read volumes corresponding to all subtasks. If the ratio of data read volumes corresponding to the four subtasks of a data table is 2:2:4:8, then the data skewness can be represented as 8:4.

[0054] In this embodiment, considering the skewness of the data table, the auxiliary diagnostic indicator of data skewness is further detected. This, combined with the operational performance indicator calculated from the data read time and data operation time, allows for more accurate screening of data tables requiring optimization or the issuance of performance alerts to users. This prevents severely skewed data tables from being screened even when they haven't been operated on, thus affecting the selection of data tables for optimization or performance alerts and improving the efficiency of data table modeling and tuning. According to this embodiment, the priority of data tables to be optimized can also be differentiated based on multiple dimensions of information, including the operational performance indicator and the auxiliary diagnostic indicator. For example, the priority weight corresponding to the operational performance indicator can be greater than the priority weight corresponding to the auxiliary diagnostic indicator.

[0055] Corresponding to the above-mentioned data operation task involving multiple sub-tasks, this specification also provides an implementation method for calculating data skewness. Specifically, calculating the data skewness of the data table to be tested based on the data read volume corresponding to the multiple sub-tasks includes:

[0056] From the multiple subtasks, determine the first target subtask with the largest data reading volume;

[0057] The data skewness of the data table to be detected is obtained by calculating the ratio of the data read volume corresponding to the first target subtask to the average data read volume corresponding to the multiple subtasks.

[0058] For example, in the above example, the ratio of data read volume corresponding to the four sub-tasks of a data table is 2:2:4:8. According to this embodiment, the data skewness can be expressed as 8:4.

[0059] In this embodiment, the data skewness is represented by the ratio of the first target subtask with the largest data reading volume to the average data reading volume of multiple subtasks. This normalizes the data skewness, so that even if the number of subtasks included in the data operation tasks of different data tables is different, the data skewness that can be directly used for comparison can be obtained, making it convenient to use the data skewness directly to filter the data tables to be optimized.

[0060] Corresponding to the above embodiments, since a data operation task of a data table includes multiple subtasks, and these subtasks are processed in parallel, the subtask with the longest data reading time encompasses the data reading time of the other subtasks. Therefore, the subtask with the longest data reading time can be used to represent the data reading time of the entire data table. By calculating the ratio of this ratio to the sum of the time spent in the data reading phase and the operation phase on the read data of the data operation task (i.e., the time spent in the data operation process of the data table under test), the operational performance index of the data table under test can be obtained. Specifically, obtaining the data reading time and the time spent in the data operation process of the data table under test includes:

[0061] From the multiple subtasks, determine the second target subtask with the longest data reading time;

[0062] The data reading time of the second target subtask and the time spent by the data table to be detected during the data operation process are obtained.

[0063] The determination of the operational performance indicators of the data table to be tested based on the data reading time and the data operation process time includes:

[0064] The ratio of the data reading time of the second target subtask to the time spent by the data table under test in the data operation process is calculated to obtain the operation performance index of the data table under test.

[0065] For example, the data reading time for the second target subtask of the data table under test is 2ns. The time spent on data operation of the data table under test is the sum of the data reading time of the second target subtask (2ns) and the time spent on performing the operation on the data (3ns), which is 5ns. Therefore, the operational performance index of the data table under test is 2 / 5.

[0066] In this embodiment, considering that the overall operation time in a database is often determined by the longest time of a subtask, the data reading time of each subtask of the data table to be tested, i.e., each data shard, is obtained. The maximum data reading time is determined from this, and it is compared with the overall time of the data operation process to obtain the operation performance index, which can accurately represent the operation performance of the data table to be tested.

[0067] It should be noted that the operational performance indicators of the data table to be tested in the methods provided in the embodiments of this specification can be used to filter data tables to be optimized, or to select data tables to issue performance alarm prompts to users, or any one or more other application scenarios. Taking the filtering of data tables to be optimized as an example, the methods provided in the embodiments of this specification may further include:

[0068] Based on the operational performance indicators of each of the data tables to be tested, a data table to be optimized is selected from the multiple data tables to be tested.

[0069] In this embodiment, the operational performance indicators calculated based on the data reading time and the data operation process time can accurately identify the data tables that affect the database operation performance, reduce the complexity of user tuning, and reduce the blindness of tuning.

[0070] To further narrow down the scope of the data tables to be optimized and reduce the randomness of optimization, in one or more embodiments of this specification, auxiliary diagnostic indicators can be combined to screen the data tables to be optimized. Specifically, the method further includes: obtaining auxiliary diagnostic indicators of the data tables to be tested. Correspondingly, the step of screening the data tables to be optimized from multiple data tables to be tested based on the operational performance indicators includes:

[0071] Based on the operational performance indicators and the auxiliary diagnostic indicators, a data table to be optimized is selected from multiple data tables to be tested.

[0072] The auxiliary diagnostic indicators include at least one of the following: data skewness, number of operations, operation time, maximum data reading volume of subtasks in the data operation task, and total data reading volume during the operation.

[0073] For example, the data table to be optimized can be selected from multiple data tables to be tested based on the operational performance indicators and the data skewness.

[0074] For example, the data table to be optimized can be selected from multiple data tables to be tested based on the operation performance indicators and the number of operations performed on the data table to be tested.

[0075] For example, the data table to be optimized can be selected from multiple data tables to be tested based on the operation performance indicators, the number of operations of the data table to be tested, and the data skewness of the data table to be tested.

[0076] It is understandable that for data tables with fewer operations, operation times outside the time range of concern, smaller maximum data read volume in subtasks of data operation tasks, lower data skewness, and / or lower total data read volume during operations, their impact on database performance is correspondingly smaller. Therefore, combining these auxiliary diagnostic indicators to screen data tables to be optimized can further narrow down the range of data tables to be optimized.

[0077] In this embodiment, by combining various auxiliary diagnostic indicators, data tables that have a significant impact on operational performance can be effectively screened out, thereby improving operational performance. For example, by combining indicators such as data skewness and query count, which reflect the distribution characteristics of the data table itself, the impact of data skewness can be combined with various indicators in the operation process, which can more accurately screen out data tables that need to be optimized.

[0078] Taking the selection of a data table to be optimized from multiple data tables to be tested based on the operational performance indicators, the number of operations, and the data skewness as an example, the range of data tables to be optimized can be narrowed down through a layer-by-layer filtering process. Specifically, for example, the selection of a data table to be optimized from multiple data tables to be tested based on the operational performance indicators and the auxiliary diagnostic indicators includes:

[0079] From multiple data tables to be tested, data tables whose number of operations meets the preset operation number condition are selected to obtain a candidate set of data tables to be optimized;

[0080] From the candidate set, data tables whose data skewness meets the preset skewness condition are selected to obtain an updated candidate set of data tables to be optimized;

[0081] From the updated candidate set, the data table whose operational performance indicators meet the preset operational performance indicator conditions is identified as the data table to be optimized.

[0082] The preset operation count condition, preset tilt condition, and preset operation performance index condition can be set according to the needs of the implementation scenario, and this specification does not impose any restrictions on them. For example, the data tables can be sorted in descending order according to the values ​​of the operation count, data tilt, and operation performance index to select the top preset number of data tables and determine the data tables to be optimized; or, for another example, a preset threshold can be set to select data tables that are greater than or equal to the preset threshold and determine the data tables to be optimized.

[0083] For example, you can first sort the data tables to be tested in descending order of the number of operations, and then select the top 500 data tables. From these 500 data tables, select those with a data skewness greater than 3. Finally, from the data tables with a data skewness greater than 3, select those with a ratio of data reading time to data operation time (i.e., operation performance index) greater than 0.5, and use these as the final data tables to be optimized.

[0084] In this embodiment, based on information from three dimensions—operational performance indicators, number of operations, and data skewness—the range of data tables to be optimized is gradually narrowed down through a layer-by-layer screening process. This approach aligns with the degree to which the three dimensions of information affect the operational performance of the data tables, achieving a scheme that accurately selects the data tables to be optimized from each of the data tables to be tested, progressing from coarse screening to fine screening.

[0085] For the selected data table to be optimized, the data distribution can be changed by reselecting specified fields, by adjusting storage nodes, or through other methods. This specification does not limit the scope of optimization. For example, in one or more embodiments of this specification, the method may further include:

[0086] For the data table to be optimized, a specified field is reselected. The specified field is used to split the data of the data table into multiple data shards, which are then distributed and stored on multiple storage nodes.

[0087] The specified field can be set by the user or automatically by the system. For example, a system using the data table performance testing method can provide a user interface through which the user can input the reselected specified field. As another example, a system using the data table performance testing method can identify fields with relatively uniform data distribution as specified fields by predicting data skewness or similar methods.

[0088] In this embodiment, a hash calculation can be performed based on the reselected specified field, and the storage node can be determined according to the hash value. The data table data can be split into multiple data shards and distributed across multiple storage nodes, thereby making the data distribution more uniform and helping to improve data operation performance.

[0089] In one or more embodiments of this specification, to facilitate screening, data operation information is obtained through database operation logs, and operation performance indicators and various auxiliary diagnostic indicators are stored in a structured manner in a diagnostic data table. Specifically, obtaining the data operation information of the data table to be tested includes:

[0090] Parse the database operation logs to obtain data operation information for multiple tables to be tested in the database;

[0091] The method further includes:

[0092] The table identifier, data operation task identifier, operation performance index, and auxiliary diagnostic index of the data table to be tested are stored in various fields of the diagnostic data table. The auxiliary diagnostic index includes at least one of the following: data skewness, number of operations, operation time, maximum data read volume of subtasks within the data operation task, and total data read volume during the operation. For example, the diagnostic data table may include a table identifier field, a data operation task identifier field, an operation performance index field, a data skewness field, etc. Correspondingly, the step of selecting the data table to be optimized from multiple data tables to be tested based on the operation performance index and the auxiliary diagnostic index includes:

[0093] By utilizing the table identifier, data operation task identifier, operation performance index, and auxiliary diagnostic index of the data table to be tested stored in each field of the diagnostic data table, as well as the preset filtering conditions corresponding to the operation performance index and auxiliary diagnostic index, the data table to be optimized is selected from multiple data tables to be tested.

[0094] For example, a general database query statement such as "select" can be used to filter the data tables to be optimized. In the "select" statement, preset filtering conditions corresponding to the operational performance indicators and auxiliary diagnostic indicators are used to select the data tables to be tested that meet the preset filtering conditions as optimized data tables. The preset filtering conditions can be set according to the actual application scenario. For example, in one or more embodiments of this specification, three rounds of filtering are performed on multiple data tables to be tested, including: first, sorting the multiple data tables to be tested from most to least queryed, and selecting the top M data tables; then, sorting the top M data tables from largest to smallest data skewness, and selecting the top N data tables; finally, from the top N data tables, selecting the data tables whose operational performance indicators are greater than or equal to a preset operational performance indicator threshold as the data tables to be optimized. In the above filtering process, the filtering results can be represented by table identifiers, such as table names. In this embodiment, the preset filtering condition for the auxiliary diagnostic indicator of operation count is: sorted by operation count from most to least, and the top M items are selected; the preset filtering condition for the auxiliary diagnostic indicator of data skewness is: sorted by data skewness from largest to smallest, and the top N items are selected; the preset filtering condition for the operation performance indicator is: greater than or equal to a preset operation performance indicator threshold. Where N and M are positive integers.

[0095] The following combination Figure 3This document provides an exemplary description of the application of a data table performance testing method provided in one embodiment of this specification in a database query scenario. Specifically, Figure 3 This diagram illustrates the implementation architecture of a data table performance testing method provided in this specification within a query application scenario. Figure 3 As shown, the implementation architecture includes: a database instance, a log collection module, a parsing and saving module, and a diagnostic module.

[0096] In this context, a database instance can be understood as a database provided to users. Within the database instance, data operations such as queries generate real-time database operation logs. For example, for a query, the database operation log can be understood as the query execution log. These logs are collected by the log collection module and input into the parsing and saving module. The parsing and saving module parses the database operation logs to obtain data operation information for multiple tables to be tested within the database. Using this information, it determines the operational performance indicators and auxiliary diagnostic indicators for each table, such as data skewness, query count, query submission time, and / or the total amount of data read during the query. The parsing and saving module saves the table identifier, query task identifier, operational performance indicators, and auxiliary diagnostic indicators (such as data skewness) of each table to be tested to a diagnostic data table. For example, the diagnostic data table can be a table within an analytical database. When it is necessary to filter tables for optimization, the diagnostic module can read relevant information from the diagnostic data table, select the tables to be optimized, and finally output diagnostic results containing the tables to be optimized.

[0097] The log collection module collects database operation logs from within the database. These logs can contain various data operation information. For example, the module can collect database operation logs for queries and other data operations from within the distributed database system. This logs include detailed information about the query process, such as query time, memory consumption, data read volume, and distributed subtasks. Furthermore, the database operation logs can also be sent to an external storage system for storage.

[0098] The parsing and saving module can parse the database operation logs using general ETL (Extract-Transform-Load) data cleaning techniques. The parsed data operation information can include: query process time, table names related to the query task, data volume and time consumed during the data reading phase of the query task, etc. Through the calculations provided by the methods in the embodiments of this specification, for a test data table, operational performance indicators and auxiliary diagnostic indicators such as data skewness can be obtained based on the data reading time and query process time. As needed, some data operation information, operational performance indicators, and auxiliary diagnostic indicators are saved to the following diagnostic data table.

[0099]

[0100]

[0101] Diagnostic data sheet

[0102] The diagnostic data table above may include:

[0103] The table identifier field can be understood as the table name field in the table above, and is used to store the table name;

[0104] The query task identifier field can be understood as the query identifier field in the table above. It is used to store the query identifier and can be combined with the table name to uniquely distinguish the query tasks of each data table.

[0105] The query submission time field stores the time the query was submitted. Using the query submission time allows you to select the time range for filtering, reducing the amount of data needed and improving filtering efficiency.

[0106] The "Data Volume" field in the query table can be understood as the "Total Rows Read" field in the table above, used to store the total number of rows read from the data table. This field can be used to filter data tables with a small total number of rows. Since data tables with a total number of rows below a certain threshold are generally considered to have a smaller impact on the overall database performance, the total number of rows read can be used to filter data tables for optimization.

[0107] The data skew field stores the degree of data skew in a data table. When reading data from different data shards, the ratio of the maximum number of rows to the average number of rows is used as the data skew. Since data tables with low data skew are generally considered to have a smaller impact on overall database performance, they can be used to filter data tables for optimization.

[0108] The Operation Performance Metrics field stores the percentage of the total query time spent reading data from the subqueries within a query task. Since tables with a low percentage have minimal impact on overall database performance, they can be used to filter tables for optimization.

[0109] First, an illustrative explanation is provided of the implementation method for calculating data skewness in the parsing and saving module within a query application scenario. In a query application scenario, the amount of data read during the execution of each subtask in a query task can be obtained based on the database operation logs. For example, the amount of data read can be represented by the number of data rows. By processing the number of data rows read by each subtask of a query task, the data skewness of the data table can be obtained. Specifically, as... Figure 4 The flowchart shown illustrates the data skewness calculation process. The data skewness calculation process for a single data table to be tested may include:

[0110] Step 402: Extract the number of data rows read by a query task for multiple subtasks of a data table under test from the database operation log.

[0111] Step 404: Determine the maximum number of data lines read for each subtask and the average number of data lines read.

[0112] Step 406: Determine whether the maximum number of data rows read is greater than or equal to the preset number of rows threshold, and whether the ratio of the maximum number of data rows read to the average number of data rows read is greater than or equal to the preset data skewness threshold.

[0113] Step 408: If so, the ratio of the maximum number of data rows read to the average number of data rows read is taken as the data skewness of the data table to be tested.

[0114] Step 410: Store the data skewness of the data table to be tested in the diagnostic data table, corresponding to the table name of the data table to be tested and the identifier of the query task.

[0115] For example: Suppose the subtask with the most data rows reads 20,000 rows, and the average number of data rows read by multiple subtasks is 2,500. The preset data skewness threshold is 3. Since the data skewness is equal to 20,000 divided by 2,500, the result is 8, which is greater than the preset data skewness threshold of 3. Therefore, the data skewness of the data table to be tested (8) is stored in the diagnostic data table along with the table name and the identifier of the query task. This table is then used as a candidate for optimization and continues to participate in the screening based on operational performance indicators to determine whether the data table needs optimization.

[0116] Next, an exemplary implementation of the method for calculating operational performance metrics in the parsing and saving module within a query application scenario will be provided. In a query application scenario, the data read time of each of the multiple subtasks within a query task can be obtained based on the database operation logs. The maximum data read time is identified, and the ratio of this maximum data read time to the overall time of the query task is calculated, thus yielding the operational performance metric. Specifically, as... Figure 5 The flowchart shown illustrates the operational performance index calculation process. The operational performance index calculation process for a single data table to be tested may include:

[0117] Step 502: Extract the data reading time of multiple subtasks of a query task for a test data table from the database operation log.

[0118] Step 504: Determine the longest data read time from the data read times of multiple subtasks.

[0119] Step 506: Calculate the ratio of the maximum data reading time to the overall time of the query task to obtain the operation performance index.

[0120] Step 508: Store the operational performance indicators of the data table to be tested, along with the table name and the identifier of the query task, in the diagnostic data table.

[0121] In this computational process, for distributed databases, the overall query time is determined by the maximum time of the subtasks. Therefore, the ratio of the maximum data read time to the overall query time is used as an operational performance indicator and stored in a diagnostic data table as a condition for selecting data tables to be optimized.

[0122] The following is an exemplary description of the process by which the diagnostic module filters the data table to be optimized based on the diagnostic data table in one or more embodiments of this specification. Exemplarily, the diagnostic module filters the final data table to be optimized through three rounds of filtering, including:

[0123] First round of screening: Sort the number of queries involved in each data table to be tested in the diagnostic data table, and select the top M data tables to be tested to enter the second round of screening. M is a whole tree with a value greater than zero. The specific value can be set according to the implementation needs.

[0124] Second round of screening: For the data tables to be tested that have entered the second round of screening, determine whether the data skewness of the data tables to be tested is greater than or equal to the preset data skewness threshold, and whether the number of data rows read by the subtask with the largest number of data rows is greater than or equal to the preset number of rows threshold. If so, the data tables to be tested will enter the third round of screening.

[0125] Third round of screening: For the data tables to be tested that have entered the third round of screening, a final screening will be conducted based on the operational performance indicators of the data tables to be tested.

[0126] Specifically, such as Figure 6 The diagram shown illustrates the three-round screening process. The three-round screening process for the data table to be tested may include:

[0127] Step 602: Sort each data table to be tested in the diagnostic data table from most to least number of queries, and select the top M data tables to be tested, where M is a positive integer.

[0128] In this step, filtering based on query count is used because if some tables have not been queried or have a low query count, it indicates that these tables do not require optimization at this time. This allows the detection effort to be focused on tables with higher query counts, improving detection efficiency. For example, this filtering step can use a combination of grouping, aggregation, and sorting queries to retrieve the top M tables to be tested from the diagnostic data table. It should be noted that to narrow the search scope of the tables to be optimized, other auxiliary diagnostic indicators can be added, such as specifying the time interval of query submission time, to constrain the query range and further narrow the search scope, improving filtering efficiency. For example, when the diagnostic data table is an SQL database table, based on the above diagnostic data table, an SQL statement such as "select" can be used for filtering in step 602, where sorting by query count from highest to lowest can be done using an SQL statement such as "desc".

[0129] Step 604: Sort the first M data tables to be tested in descending order of data skewness, and select the first N data tables to be tested, where N is a positive integer less than M.

[0130] After initial filtering of the table based on query count, this step further filters based on data skewness. For example, if the diagnostic data table is an SQL database table, step 604 can be filtered using an SQL statement like "select," and sorting by data skewness from largest to smallest can be done using an SQL statement like "desc." Step 604 also uses the table names filtered in step 602 as filtering conditions, such as using an expression like "(table_name in('xxx','xxx'))," where ('xxx','xxx') represents the range of table names filtered in step 602. Through step 604, the data table to be tested, filtered by query count, can be further filtered based on the dimension of data skewness.

[0131] Step 606: From the first N data tables to be tested, select the data tables whose operational performance indicators are greater than or equal to the preset operational performance indicator thresholds to obtain the final data tables to be optimized.

[0132] After further filtering based on data skewness, this step involves final filtering based on operational performance metrics determined by the time spent in the data reading phase and the overall query time. For example, if the diagnostic data table is an SQL database table, the "select" SQL statement can be used for filtering in step 606, using the table names filtered in step 604 as filtering conditions. Step 606 primarily uses the ratio of the time spent in the data reading phase to the overall query time for each data shard of the table under test—that is, the operational performance metric—for filtering. For example, if the operational performance metric is greater than 0.5, it indicates that the impact of data skew exceeds half of the query time; such a table is considered for optimization, and optimization will result in a significant improvement in query performance.

[0133] It should be noted that the above SQL statements are only used to illustrate the processing procedures provided in the embodiments of this specification, and do not constitute a limitation on the methods provided in the embodiments of this specification. Depending on the implementation scenario, any query statement can be used for filtering.

[0134] As can be seen from the above processing, the data table performance testing method provided in the embodiments of this specification filters the data tables to be optimized in three stages, performing multi-dimensional filtering on the data tables to be tested. This provides users with the data tables that need optimization, reducing the threshold for users' data modeling and tuning. To improve filtering efficiency, the table identifier, query task identifier, operation performance indicators, and auxiliary diagnostic indicators of each data table to be tested can be gradually stored in the respective fields of the diagnostic data table for filtering purposes. For example, when cleaning database operation logs, data operation information of the tables to be tested can be extracted from the logs, such as query task identifiers, table names, and query submission times. This information is then formatted and saved to a diagnostic data table. Based on the information already saved in the diagnostic data table, the tables are filtered according to the number of queries. For the tables to be tested selected based on the number of queries, the data skewness is calculated and saved to the diagnostic data table. Based on the data skewness, a candidate set of tables to be optimized is further selected. Then, operational performance indicators are calculated for the tables to be tested in the candidate set to determine the impact of data skew on queries. These operational performance indicators are saved to the diagnostic data table, and based on these indicators, tables that need optimization are further selected.

[0135] Corresponding to the above method embodiments, this specification also provides embodiments of a data table performance testing device. Figure 7 A schematic diagram of a data table performance testing device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0136] The information acquisition module 702 is configured to acquire data operation information of the data table to be tested.

[0137] The time acquisition module 704 is configured to use the data operation information of the data table to be tested to acquire the data reading time and the data operation process time of the data table to be tested.

[0138] The performance determination module 706 is configured to determine the operational performance indicators of the data table to be tested based on the data reading time and the data operation process time.

[0139] In one or more embodiments of this specification, the apparatus further includes:

[0140] The operation task determination module is configured to determine the data operation task of the data table to be tested using the data operation information of the data table to be tested.

[0141] The data read volume determination module is configured to obtain the data read volume corresponding to multiple subtasks in the data operation task. The data table to be detected is divided into multiple data shards, which are stored in multiple storage nodes. One of the subtasks is used to read data from one of the storage nodes and perform data operations.

[0142] The tilt calculation module is configured to calculate the data tilt of the data table to be detected based on the data reading volume corresponding to the multiple sub-tasks.

[0143] In one or more embodiments of this specification, the apparatus further includes:

[0144] The optimization table filtering module is configured to filter out the data table to be optimized from multiple data tables to be tested based on the operational performance indicators.

[0145] In one or more embodiments of this specification, the optimization table filtering module is configured to filter out a data table to be optimized from a plurality of data tables to be tested based on the operation performance indicators and the auxiliary diagnostic indicators; the auxiliary diagnostic indicators include at least one of the following: data skewness, number of operations, operation time, maximum data reading volume of subtasks in the data operation task, and total data reading volume during the operation.

[0146] In one or more embodiments of this specification, the optimized table filtering module includes:

[0147] The operation count filtering submodule is configured to filter out data tables whose operation counts meet preset operation count conditions from each of the data tables to be tested, thereby obtaining a candidate set of data tables to be optimized.

[0148] The skewness filtering submodule is configured to filter out data tables whose data skewness meets preset skewness conditions from the candidate set of the data tables to be optimized, and obtain an updated candidate set of data tables to be optimized.

[0149] The performance index filtering submodule is configured to determine, from the updated candidate set, the data table whose operational performance index meets the preset operational performance index conditions, as the data table to be optimized.

[0150] In one or more embodiments of this specification, the tilt calculation module includes:

[0151] The data maximum value filtering submodule is configured to determine the first target subtask with the largest corresponding data reading volume from the multiple subtasks.

[0152] The tilt calculation submodule is configured to calculate the ratio of the data read volume corresponding to the first target subtask to the average of the data read volumes corresponding to the multiple subtasks, thereby obtaining the data tilt of the data table to be detected.

[0153] In one or more embodiments of this specification, the time consumption acquisition module includes:

[0154] The maximum time consumption filtering submodule is configured to determine the second target subtask with the longest data reading time from the multiple subtasks.

[0155] The time consumption acquisition submodule is configured to acquire the data reading time of the second target subtask and the time consumed by the data table to be detected during the data operation process.

[0156] The performance determination module is configured to calculate the ratio of the data reading time of the second target subtask to the time spent by the data table under test in the data operation process, thereby obtaining the operation performance index of the data table under test.

[0157] In one or more embodiments of this specification, the information acquisition module is configured to parse database operation logs to obtain data operation information of multiple data tables to be tested in the database. The device further includes an indicator storage module, configured to store the table identifier, data operation task identifier, operation performance indicators, and auxiliary diagnostic indicators of each data table to be tested in various fields of a diagnostic data table. The auxiliary diagnostic indicators include: data skewness, number of operations, operation time, maximum data read volume of subtasks in the data operation task, and / or total data read volume during the operation. The optimization table filtering module is configured to use the diagnostic data table to filter out data tables to be optimized from the data tables to be tested.

[0158] In one or more embodiments of this specification, the apparatus further includes: a distribution optimization module configured to reselect a specified field for the data table to be optimized, the specified field being used to split the data of the data table into multiple data shards, the multiple data shards being distributed and stored on multiple storage nodes.

[0159] The above is a schematic scheme of a data table performance testing device according to this embodiment. It should be noted that the technical solution of this data table performance testing device and the technical solution of the data table performance testing method described above belong to the same concept. For details not described in detail in the technical solution of the data table performance testing device, please refer to the description of the technical solution of the data table performance testing method described above.

[0160] Corresponding to the above method embodiments, this specification also provides embodiments of a data table performance testing system. Figure 8 A schematic diagram of a data table performance testing system according to one embodiment of this specification is shown. Figure 8 As shown, the system includes: a detection node 802, a database service node 804, a computing node 806, and multiple storage nodes 808.

[0161] The detection node 802 is configured to detect the operational performance indicators of the data tables in the database using the data table performance detection method as described in any embodiment of this specification.

[0162] The database service node 804 is configured to run the database and issue data operation tasks of the database to the computing nodes.

[0163] The computing node 806 is configured to perform data manipulation tasks on the data tables of the database.

[0164] The storage node 808 is configured to store data shards of the data table. The data shards are obtained by splitting the data of the data table using a specified field of the data table, and the multiple data shards of the data table are distributed and stored on multiple storage nodes.

[0165] The above is an illustrative scheme of a data table performance testing system according to this embodiment. It should be noted that the technical solution of this data table performance testing system and the technical solution of the data table performance testing method described above belong to the same concept. Details not described in detail in the technical solution of the data table performance testing system can be found in the description of the technical solution of the data table performance testing method described above.

[0166] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0167] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0168] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0169] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0170] The processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data table performance detection method.

[0171] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data table performance testing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data table performance testing method described above.

[0172] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data table performance detection method.

[0173] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the data table performance testing method described above. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data table performance testing method described above.

[0174] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data table performance testing method.

[0175] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned data table performance testing method belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the aforementioned data table performance testing method.

[0176] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0177] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0178] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for testing the performance of a data table, comprising: Obtain data manipulation information from the data table to be tested; Using the data operation information, the data reading time and the total time of the data operation process of the data table to be tested are obtained. Based on the data reading time and the data operation process time, the operation performance index of the data table to be tested is determined. According to the operation performance index and auxiliary diagnostic index, the data table to be optimized is selected from multiple data tables to be tested. The auxiliary diagnostic index includes at least one of the following: data skewness, number of operations, operation time, maximum data reading volume of subtasks in the data operation task, and total data reading volume in the operation. The data table to be optimized indicates the data table with data skew problem and needs to be optimized.

2. The method according to claim 1, further comprising: Based on the operational performance indicators, a data table to be optimized is selected from multiple data tables to be tested.

3. The method according to claim 2, further comprising: Obtain auxiliary diagnostic indicators from the data table to be tested; The step of selecting the data table to be optimized from multiple data tables to be tested based on the operational performance indicators includes: Based on the operational performance indicators and the auxiliary diagnostic indicators, a data table to be optimized is selected from multiple data tables to be tested. The auxiliary diagnostic indicators include at least one of the following: data skewness, number of operations, operation time, maximum data reading volume of subtasks in the data operation task, and total data reading volume during the operation.

4. The method according to claim 3, wherein the auxiliary diagnostic indicators include the number of operations and data skewness, and the step of selecting the data table to be optimized from multiple data tables to be tested based on the operational performance indicators and the auxiliary diagnostic indicators includes: From multiple data tables to be tested, data tables whose number of operations meets the preset operation number condition are selected to obtain a candidate set of data tables to be optimized; From the candidate set, data tables whose data skewness meets the preset skewness condition are selected to obtain an updated candidate set of data tables to be optimized; From the updated candidate set, the data table whose operational performance indicators meet the preset operational performance indicator conditions is identified as the data table to be optimized.

5. The method according to claim 3, wherein the auxiliary diagnostic indicator includes data skewness, and the step of obtaining the auxiliary diagnostic indicator of the data table to be tested includes: The data operation tasks of the data table to be tested are determined using the data operation information of the data table to be tested. The data read volume corresponding to multiple subtasks in the data operation task is obtained. The data table to be detected is divided into multiple data shards. The multiple data shards are stored in multiple storage nodes. One of the subtasks is used to read the data of one of the data shards from one of the storage nodes and perform data operations. Based on the data reading volume corresponding to the multiple sub-tasks, the data skewness of the data table to be detected is calculated.

6. The method according to claim 5, wherein calculating the data skewness of the data table to be detected based on the data read volume corresponding to the plurality of subtasks includes: From the multiple subtasks, determine the first target subtask with the largest data reading volume; The data skewness of the data table to be detected is obtained by calculating the ratio of the data read volume corresponding to the first target subtask to the average data read volume corresponding to the multiple subtasks.

7. The method according to claim 6, wherein obtaining the data reading time and the data operation time of the data table to be detected during the data operation process includes: From the multiple subtasks, determine the second target subtask with the longest data reading time; The data reading time of the second target subtask and the time spent by the data table to be detected during the data operation process are obtained. The determination of the operational performance indicators of the data table to be tested based on the data reading time and the data operation process time includes: The ratio of the data reading time of the second target subtask to the time spent by the data table under test in the data operation process is calculated to obtain the operation performance index of the data table under test.

8. The method according to claim 3, wherein obtaining the data operation information of the data table to be detected includes: Parse the database operation logs to obtain data operation information for multiple tables to be tested in the database; The method further includes: The table identifier, data operation task identifier, operation performance index, and auxiliary diagnostic index of the data table to be tested are stored in each field of the diagnostic data table. The step of selecting the data table to be optimized from multiple data tables to be tested based on the operational performance indicators and the auxiliary diagnostic indicators includes: By utilizing the table identifier, data operation task identifier, operation performance index, and auxiliary diagnostic index of the data table to be tested stored in each field of the diagnostic data table, as well as the preset filtering conditions corresponding to the operation performance index and auxiliary diagnostic index, the data table to be optimized is selected from multiple data tables to be tested.

9. The method according to claim 3, further comprising: For the data table to be optimized, a specified field is reselected. The specified field is used to split the data of the data table into multiple data shards, which are then distributed and stored on multiple storage nodes.

10. A data table performance testing system, comprising: Detection nodes, database service nodes, computing nodes, and multiple storage nodes; The detection node is configured to use the data table performance detection method as described in any one of claims 1-9 to detect the operational performance indicators of the data tables in the database; The database service node is configured to run the database and issue data operation tasks of the database to the computing node; The computing node is configured to perform data manipulation tasks on the data tables of the database; The storage node is configured to store data shards of the data table. The data shards are obtained by splitting the data of the data table using a specified field of the data table, and the multiple data shards of the data table are stored separately on multiple storage nodes.

11. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the data table performance detection method according to any one of claims 1-9.

12. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data table performance testing method according to any one of claims 1-9.

Images