Data management method and related device

Abstract

The embodiment of the invention provides a data governance method and a related device, and the method comprises the steps: obtaining log information in a database through connection information based on the database; checking the data corresponding to the log information based on a risk checking rule and the log information, and determining risk data in the data; and treating the risk data to obtain a treatment result. Wherein the risk checking rule comprises one or more of a low-performance data checking rule, a high-risk session checking rule, a memory occupation checking rule, an execution plan risk checking rule and an online checking rule. According to the method provided by the embodiment of the invention, low-performance data, high-risk sessions, data with high memory occupation and execution plans with risks can be checked on the basis of the risk checking rules and the log information, so that full-life-cycle data management is realized, and the stability of a database is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data governance method and related devices. Background Art

[0002] Database is a way to store and organize data, which can make operations such as adding, accessing, updating and deleting data more convenient and efficient. However, with the increasing application of database, various data problems have emerged, which may affect the stability of the database.

[0003] Currently, we can detect the execution time of data, identify data with execution time exceeding the preset threshold as risky data, and manage it. However, this method can only identify data with execution time exceeding the preset threshold, but cannot identify other types of risky data, making it difficult to ensure the stability of the database.

[0004] Therefore, a solution is urgently needed to solve the above technical problems. Summary of the invention

[0005] Based on the above problems, the present application provides a data governance method and related devices.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] First aspect: The present application embodiment provides a data governance method, including:

[0008] Based on the connection information of the database, obtaining log information in the database;

[0009] Based on the risk screening rules and the log information, each data corresponding to the log information is screened to determine the risk data in the data; the risk screening rules include one or more of low-performance data screening rules, high-risk session screening rules, memory crowding screening rules, execution plan risk screening rules, and online inspection rules;

[0010] The risk data is governed to obtain a governance result.

[0011] In a possible implementation, when the risk screening rule includes a low-performance data screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes:

[0012] Based on the log information, determine, from each data corresponding to the log information, first data to be checked whose execution time is greater than a first threshold and whose total processor time consumed when executing the data ranks in the top N; wherein N is an integer greater than 0;

[0013] Based on the low-performance data screening rule and the log information, the first data to be screened is screened to determine risk data among the data.

[0014] In a possible implementation, when the risk screening rule includes a high-risk session screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes:

[0015] Based on the log information, determining, from each data corresponding to the log information, a session whose execution duration is greater than a second threshold;

[0016] Based on the high-risk session screening rule and the log information, the session is screened to determine the risk data among the data.

[0017] In a possible implementation, when the risk screening rule includes a memory occupation screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes:

[0018] Based on the memory occupation troubleshooting rule and the log information, data corresponding to the log information that occupies a memory greater than a third threshold is determined as risk data.

[0019] In a possible implementation, when the risk screening rule includes an execution plan risk screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes:

[0020] Based on the log information, determine, from each data corresponding to the log information, second data to be checked whose execution time is greater than a fourth threshold and whose total processor time consumed when executing the data ranks in the top M positions; wherein M is an integer greater than 0;

[0021] Determining text data corresponding to the second data to be checked based on the information identifier of the second data to be checked;

[0022] Based on the execution plan risk screening rules and the log information, the text data is screened to determine the risk data in the data.

[0023] In a possible implementation, when the risk screening rule includes an online inspection rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes:

[0024] Based on the online inspection rules and the log information, a database corresponding to the log information is scanned to determine the risk data in the data.

[0025] In a possible implementation, the managing the risk data to obtain a management result includes:

[0026] Storing the risk data in a risk data list;

[0027] In response to a user's modification operation on risk data in the risk data list, obtaining governed data;

[0028] Based on the managed data, the risk data list is updated to obtain a management result.

[0029] Second aspect: The embodiment of the present application provides a data management device, including:

[0030] Acquisition unit, troubleshooting unit, and governance unit;

[0031] The acquisition unit is used to acquire the log information in the database based on the connection information of the database;

[0032] The troubleshooting unit is used to troubleshoot each data corresponding to the log information based on the risk troubleshooting rules and the log information, and determine the risk data in the data; the risk troubleshooting rules include one or more of low-performance data troubleshooting rules, high-risk session troubleshooting rules, memory crowding troubleshooting rules, execution plan risk troubleshooting rules, and online inspection rules;

[0033] The governance unit is used to govern the risk data and obtain governance results.

[0034] In a possible implementation, when the risk screening rule includes a low-performance data screening rule, the screening unit is specifically configured to:

[0035] Based on the log information, determine, from each data corresponding to the log information, first data to be checked whose execution time is greater than a first threshold and whose total processor time consumed when executing the data ranks in the top N; wherein N is an integer greater than 0;

[0036] Based on the low-performance data screening rule and the log information, the first data to be screened is screened to determine risk data among the data.

[0037] In a possible implementation, when the risk screening rule includes a high-risk session screening rule, the screening unit is specifically configured to:

[0038] Based on the log information, determining, from each data corresponding to the log information, a session whose execution duration is greater than a second threshold;

[0039] Based on the high-risk session screening rule and the log information, the session is screened to determine the risk data among the data.

[0040] In a possible implementation, when the risk screening rule includes a memory occupation screening rule, the screening unit is specifically configured to:

[0041] Based on the memory occupation troubleshooting rule and the log information, data corresponding to the log information that occupies a memory greater than a third threshold is determined as risk data.

[0042] In a possible implementation, when the risk screening rule includes an execution plan risk screening rule, the screening unit is specifically configured to:

[0043] Based on the log information, determine, from each data corresponding to the log information, second data to be checked whose execution time is greater than a fourth threshold and whose total processor time consumed when executing the data ranks in the top M positions; wherein M is an integer greater than 0;

[0044] Determining text data corresponding to the second data to be checked based on the information identifier of the second data to be checked;

[0045] Based on the execution plan risk screening rules and the log information, the text data is screened to determine the risk data in the data.

[0046] In a possible implementation, when the risk screening rule includes an online inspection rule, the screening unit is specifically configured to:

[0047] Based on the online inspection rules and the log information, a database corresponding to the log information is scanned to determine the risk data in the data.

[0048] In a possible implementation, the governance unit is specifically configured to:

[0049] Storing the risk data in a risk data list;

[0050] In response to a user's modification operation on risk data in the risk data list, obtaining governed data;

[0051] Based on the managed data, the risk data list is updated to obtain a management result.

[0052] A third aspect: An embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory;

[0053] The memory is used to store program code and transmit the program code to the processor;

[0054] The processor is used to execute the steps of a data governance method as described in the first aspect above according to the instructions in the program code.

[0055] Fourth aspect: An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a data governance method as described in the first aspect above are implemented.

[0056] Compared with the prior art, this application has the following beneficial effects:

[0057] The embodiment of the present application provides a data governance method and related devices, which obtains the log information in the database based on the connection information of the database; based on the risk screening rules and the log information, screens the data corresponding to the log information to determine the risk data in the data; and manages the risk data to obtain a governance result. Among them, the risk screening rules include one or more of low-performance data screening rules, high-risk session screening rules, memory crowding screening rules, execution plan risk screening rules, and online inspection rules. The method provided in the embodiment of the present application is based on the risk screening rules and the log information, and can screen low-performance data, high-risk sessions, data with high memory crowding, and execution plans with risks, so as to achieve data governance throughout the entire life cycle and ensure the stability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A flowchart of a data governance method provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of a SQL governance method provided in an embodiment of the present application;

[0061] Figure 3 A structural schematic diagram of a data management device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] Database is a way to store and organize data, which can make operations such as adding, accessing, updating and deleting data more convenient and efficient. However, with the increasing application of database, various data problems have emerged, which may affect the stability of the database.

[0064] For example, due to performance differences between different databases and differences in optimizer implementation mechanisms, during the database migration process, some data that does not pose risks in the original environment may become risky data after being migrated to the new environment. The existence of risky data is one of the important factors affecting database stability. To ensure database stability, risky data needs to be identified and managed.

[0065] Currently, we can detect the execution time of data, identify data with execution time exceeding the preset threshold as risky data, and manage it. However, this method can only identify data with execution time exceeding the preset threshold, but cannot identify other types of risky data, making it difficult to ensure the stability of the database.

[0066] Based on this, an embodiment of the present application provides a data governance method, which obtains log information in the database based on the connection information of the database; based on the risk screening rules and the log information, screens the data corresponding to the log information to determine the risk data in the data; and manages the risk data to obtain a governance result. Among them, the risk screening rules include one or more of low-performance data screening rules, high-risk session screening rules, memory crowding screening rules, execution plan risk screening rules, and online inspection rules. The method provided in the embodiment of the present application is based on the risk screening rules and the log information, and can screen low-performance data, high-risk sessions, data with high memory crowding, and execution plans with risks, so as to achieve data governance throughout the entire life cycle and ensure the stability of the database.

[0067] Combine the following Figure 1 A data governance method provided in an embodiment of the present application is introduced. The figure is a flow chart of a data governance method provided in an embodiment of the present application, including S101-S103.

[0068] S101. Obtain log information in a database based on database connection information.

[0069] The method provided in the embodiment of the present application can configure database connection information of different environments in the system to connect to the database corresponding to each environment.

[0070] In a possible implementation, according to the database connection information of each environment input by the user, the corresponding database client can be used to connect to the function test, performance test and production environment database.

[0071] To improve the security of the database, two independent databases can be used in the embodiment of the present application to connect the functional test, performance test, and production environment databases. For example, one database is connected to the functional test and performance test in the test environment, and the other database is connected to the production environment database in the production environment.

[0072] In the embodiment of the present application, after connecting to the database corresponding to each environment, log information in each database can be obtained, wherein the log information may include but is not limited to data execution information in the system table, session execution records, and execution plan information.

[0073] In a possible implementation, after connecting to the database based on the database connection information, data execution information can be obtained. Taking the data as Structured Query Language (SQL) data as an example, in the embodiment of the present application, SQL execution information in the system table can be obtained through SQL statements.

[0074] For example, when the database is an OceanBase database, all SQL execution information within one hour is captured every hour through the oceanbase.gv$sql_audit table on the user side corresponding to the database system.

[0075] The SQL execution information may include, but is not limited to, the Internet Protocol (IP) address, port number, SQL identifier (ID), the ID of the client sending the request, the name of the client sending the request, the actual SQL statement, the number of affected rows, the execution time, and whether the full table is scanned.

[0076] Based on the data execution information, the number of times the data is executed can be determined. At the same time, the average execution time corresponding to the data can be determined through the execution time in the data execution information. For example, if 5 data execution information of data A are obtained, it can be determined that data A is executed 5 times, and the average execution time corresponding to data A can be calculated based on the execution time in each data execution information.

[0077] In the embodiment of the present application, the total processor time consumed when executing the data can be determined based on the product of the number of executions of the data and its average execution time. The data is sorted based on the total processor time consumed when executing each data, so that each data can be subsequently checked based on the sorting result, thereby improving the checking efficiency.

[0078] In a possible implementation, after connecting to a database based on the database connection information, the session execution record may be obtained.

[0079] Taking the OceanBase database as an example, historical session execution information can be obtained through the oceanbase._all_virtual_processlist table on the user side corresponding to the database system. The session execution information may include but is not limited to IP address, port number, session ID, session status, SQL text and other information.

[0080] In a possible implementation, after connecting to a database based on the database connection information, the execution plan information may be obtained.

[0081] Taking the OceanBase database as an example, you can obtain the execution plan information, IP address, port number, user ID, and execution plan IP through the _all_virtual_plan_cache_plan_explain table on the user side corresponding to the database system.

[0082] S102: Based on the risk screening rules and the log information, each data corresponding to the log information is screened to determine the risk data in the data.

[0083] Among them, the risk troubleshooting rules include but are not limited to one or more of low-performance data troubleshooting rules, high-risk session troubleshooting rules, memory crowding troubleshooting rules, execution plan risk troubleshooting rules, and online inspection rules.

[0084] In a possible implementation, when the risk screening rule includes a low-performance data screening rule, based on the log information, the first data to be screened whose execution time is greater than a first threshold and whose total processor time when executing the data ranks in the top N can be determined from the data corresponding to the log information, where N is an integer greater than 0.

[0085] Exemplarily, based on the low-performance data troubleshooting rules, the log information within a time period is scanned, and according to the SQL execution information table in the log information, the SQL statements whose execution time is greater than a first threshold and whose total processor time when executing the data ranks in the top N can be determined as the first data to be troubleshooted.

[0086] The total processor time consumed when executing data can be determined based on the product of the number of executions of the data and its average execution time, and the top N SQL statements are determined in descending order of total processor time consumed.

[0087] It should be noted that the first threshold and the value of N are not specifically limited in the embodiment of the present application, and can be set according to actual needs. For example, the method provided in the embodiment of the present application can check for SQL statements whose execution time is greater than the first threshold and whose total processor time when executing the data ranks in the top 30.

[0088] After the first data to be checked is determined, the first data to be checked may be checked based on the low-performance data checking rule and the log information to determine the risk data in each data.

[0089] The low-performance data troubleshooting rules corresponding to different databases may be different. Taking the OceanBase database as an example, the low-performance data troubleshooting rules corresponding to the OceanBase database include checking whether the number of discrete conditional constant values ​​in the IN set is greater than 100; checking the data without the is not null condition in the SQL using the not in statement; and checking the risk of splicing a large number of insert...values.

[0090] In the embodiment of the present application, when the number of discrete conditions in the IN set is greater than 100, the SQL performance will drop sharply. By checking whether the number of discrete condition constant values ​​in the IN set is greater than 100, the SQL performance can be avoided. If the is not null condition is not added to the SQL using the not in statement, it may cause a logical result error. By checking the data that does not add the is not null condition in the SQL using the not in statement, the logical result error caused by not adding the isnot null condition in the SQL using the not in statement can be avoided. In the third version of OceanBase, when insert...values ​​is used to splice multiple data, the database will split the splicing results of insert...values ​​and check them one by one to ensure the correctness and integrity of the data. However, this process will cause a significant increase in memory usage when the amount of data is large. In the embodiment of the present application, by checking the amount of data spliced ​​using insert...values, a large amount of memory can be avoided.

[0091] In a possible implementation, when the risk screening rules include high-risk session screening rules, based on the log information, sessions whose execution duration is greater than a second threshold can be determined from the data corresponding to the log information; based on the high-risk session screening rules and the log information, the sessions are screened to determine the risk data in the data.

[0092] Exemplarily, based on the high-risk session screening rule, the log information is scanned, and based on the session execution information table within a time period in the log information, the sessions whose execution time is greater than the second threshold are screened to determine risky sessions.

[0093] Taking the OceanBase database as an example, the high-risk session troubleshooting rules include checking whether the number of sessions reporting 6005 lock conflicts, 4138 snapshot outdated errors, and 4026 value not found errors exceeds the threshold.

[0094] Among them, the error 6005 lock conflict usually occurs when multiple sessions try to modify the same data at the same time. When one session holds a lock and other sessions try to obtain the same lock, a conflict occurs. The error 4138 snapshot is too old in the session, which usually occurs when the queried data is modified by other sessions during the execution of the session, resulting in the current session being unable to obtain sufficiently new data. The error 4026 value not found usually means that the data referenced in the query or operation does not exist in the database. This may be caused by data deletion, update or data inconsistency.

[0095] It is understandable that the embodiment of the present application does not specifically limit the size of the quantity threshold, which can be set according to actual conditions. For example, the quantity threshold can be 10,000.

[0096] In a possible implementation, when the risk screening rule includes a memory occupation screening rule, based on the memory occupation screening rule and the log information, data corresponding to the log information that occupies memory greater than a third threshold can be determined as risk data.

[0097] In one example, based on the memory crowding troubleshooting rules and the log information, the memory crowding related table in the log information can be queried regularly, and SQL that crowds out memory greater than a third threshold can be identified according to corresponding data manipulation language (DML) statements, and determined as risky data.

[0098] In another example, based on the memory occupation troubleshooting rules and the log information, the memory occupation related table in the log information can be queried regularly, and the SQL that occupies the most memory can be identified according to the corresponding DML statements and determined as risk data.

[0099] It can be understood that in the embodiment of the present application, there is no specific limitation on the size of the third threshold, which can be set according to actual needs.

[0100] In a possible implementation, when the risk screening rule includes an execution plan risk screening rule, based on the log information, second data to be screened whose execution time is greater than a fourth threshold and whose total processor time when executing the data ranks in the top M positions can be determined from the data corresponding to the log information. Wherein, M is an integer greater than 0. Based on the information identifier of the second data to be screened, the text data corresponding to the second data to be screened is determined.

[0101] Based on the execution plan risk screening rules and the log information, the text data is screened to determine the risk data in the data.

[0102] Exemplarily, the text data corresponding to the second data to be checked may be text data in the plan information table. Taking the OceanBase database as an example, based on the text data, it is possible to check whether there are valid association conditions between the associated tables, and then determine the risk data in the data.

[0103] In a possible implementation, when the risk screening rule includes an online inspection rule, the database corresponding to the log information may be scanned based on the online inspection rule and the log information to determine the risk data in the data.

[0104] Exemplarily, the query consumption in the online inspection process is small, and the query analysis can be directly performed in the user database. In the embodiment of the present application, the various data dictionaries such as sequences, objects, partitions, etc. in the database can be scanned at intervals to determine the risk data in the data.

[0105] S103: Govern the risk data to obtain governance results.

[0106] In an embodiment of the present application, after the risk data is determined, the risk data can be stored in a risk data list; in response to a user's modification operation on the risk data in the risk data list, the governed data is obtained; based on the governed data, the risk data list is updated to obtain a governance result.

[0107] In a possible implementation, the risk data may be processed and stored in a risk data list. Exemplarily, the process of processing the risk data includes, but is not limited to, associating system information and database connection information with the risk data, determining the type of the risk data, determining optimization suggestions, adding scanning batch information, etc.

[0108] Before storing the risk data in the risk data list, the risk data can be compared with the data in the white list. If the risk data matches the data in the white list, the risk data is marked. For example, the "whether it is a white list data item" corresponding to the risk data is set to true to indicate that the risk data is data in the white list.

[0109] Users can query the risk data list to obtain relevant information about the risk data. After users check and modify the risk data, they can update the risk data list and obtain the governance results.

[0110] Exemplarily, the "Is it modified item" in the risk data list can be set to yes, indicating that the risk number in the risk data list has been modified. After uploading the current risk data list, the database can update the relevant data based on the modified data in the risk data list.

[0111] To summarize, the embodiment of the present application provides a data governance method, which, based on risk screening rules and the log information, can screen low-performance data, high-risk sessions, data with high memory usage, and risky execution plans, thereby achieving full life cycle data governance and ensuring the stability of the database.

[0112] For ease of understanding, the following example uses SQL as the data and OceanBase as the database. Figure 2 An overall introduction to the method provided in the embodiments of the present application is given.

[0113] like Figure 2 As shown, this figure is a flow chart of a SQL governance method provided in an embodiment of the present application.

[0114] S201. Configure database connection.

[0115] Configure database connection information for different environments in the system to connect the databases corresponding to each environment. To improve the security of the database, two independent databases can be used in the embodiment of the present application to connect the functional test, performance test, and production environment databases. For example, one set of databases is connected to the functional test and performance test in the test environment, and the other set of databases is connected to the production environment database in the production environment.

[0116] S202: Capture log information.

[0117] After connecting to the database, all SQL execution information within one hour is captured every hour through the oceanbase.gv$sql_audit table on the user side corresponding to the OceanBase database system.

[0118] The SQL execution information may include, but is not limited to, the IP address, port number, SQL ID, the ID of the client sending the request, the name of the client sending the request, the actual SQL statement, the number of affected rows, the execution time, and whether the full table is scanned.

[0119] The captured SQL execution information and environment information are stored in the database of the data analysis server, and the total processor time consumed when executing the data is determined based on the product of the number of executions of the data and its average execution time.

[0120] The historical session execution information can be obtained through the oceanbase._all_virtual_processlist table of the user end corresponding to the database system. The session execution information may include but is not limited to the IP address, port number, session ID, session status, SQL text, etc. The captured session execution information and environment information are stored in the database of the data analysis server.

[0121] Through the _all_virtual_plan_cache_plan_explain table of the user side corresponding to the database system, you can obtain the execution plan information, IP address, port number, user side ID and execution plan IP, etc. The captured execution plan information and environment information are stored in the database of the data analysis server.

[0122] S203. Data analysis.

[0123] In the embodiment of the present application, data analysis may include the following five aspects:

[0124] First, on the data analysis server, based on the low-performance data troubleshooting rules, scan the log information within a period of time. According to the SQL execution information table in the log information, for SQL statements whose execution time is greater than the first threshold and whose total processor time when executing the data ranks in the top 30, check whether the number of discrete conditional constant values ​​in the IN set is greater than 100; check the data without the is not null condition in the SQL using the not in statement; check the risk of splicing a large number of insert...values, etc.

[0125] Second aspect: On the data analysis server, based on the high-risk session troubleshooting rules, scan the log information, and based on the session execution information table within a period of time in the log information, for sessions whose execution time is greater than the second threshold, check whether the number of sessions reporting 6005 lock conflicts, 4138 snapshot outdated errors, and 4026 value not found errors exceeds 10,000.

[0126] The third aspect: Regularly query the memory usage-related tables in the log information, identify the SQL that occupies the most memory based on the corresponding DML statements, and identify it as a risky SQL.

[0127] Fourth aspect: On the data analysis server, based on the execution plan risk investigation rules, the execution plan information table of the log information within a certain time period is scanned, and for the SQL texts corresponding to the SQL statements whose execution time is greater than the fourth threshold and whose total processor time when executing the data ranks in the top 30, it is checked whether there are valid association conditions between the related tables, and then the risk SQL in the data is determined.

[0128] A fifth aspect: Directly query and analyze in the user database based on the online inspection rules. For example, multiple data dictionaries such as sequences, objects, and partitions in the database may be scanned at long intervals to determine the risky SQL in the data.

[0129] S204. Record risk SQL.

[0130] The risk SQL is processed and stored in the risk data list. Exemplarily, the process of processing the risk data includes, but is not limited to, associating the risk data with system information and database connection information, determining the type of the risk data, determining optimization suggestions, adding scanning batch information, and the like.

[0131] The risk SQL is stored in the risk data list, and the risk SQL can be compared with the SQL in the white list. If the risk SQL matches the SQL in the white list, the "whether it is a white list data item" corresponding to the risk SQL is set to true.

[0132] S205: Output the query results.

[0133] Users can query the risk data list according to time period to obtain relevant information of risk SQL.

[0134] Exemplarily, the query results can be output according to the scanning batches, and the information except the white list and the modified risk SQL within the query time period is uniformly output in the form of a table to obtain a risk SQL governance table.

[0135] The risk SQL management table may include, but is not limited to, system name, server and port information, SQLID, SQL text, risk SQL type, optimization suggestions, execution time, whether it has been modified, and whether it is a SQL in the whitelist.

[0136] S206. Feedback on governance results.

[0137] After the user checks and modifies the risk SQL, he can change the Modified column in the risk SQL management table to Yes, upload the risk SQL management table, and update the risk data list in the corresponding time period to obtain the management results.

[0138] To summarize, the embodiment of the present application provides a data governance method, which, based on risk screening rules and the log information, can screen low-performance data, high-risk sessions, data with high memory usage, and risky execution plans, thereby achieving full life cycle data governance and ensuring the stability of the database.

[0139] At the same time, risk data can be scanned at the source code and execution plan levels, and different thresholds can be set for monitoring in functional testing, performance testing, and production environments to timely discover risk data and determine the causes and modification suggestions, thereby improving problem troubleshooting efficiency.

[0140] This application provides a data management device, see Figure 3 This figure is a structural diagram of a data governance device provided in an embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above method, and some contents will not be repeated here.

[0141] The present application embodiment provides a data management device 1100, including:

[0142] Acquisition unit 1101, troubleshooting unit 1102 and management unit 1103;

[0143] The acquisition unit 1101 is used to acquire log information in the database based on the connection information of the database;

[0144] The troubleshooting unit 1102 is used to troubleshoot each data corresponding to the log information based on the risk troubleshooting rules and the log information, and determine the risk data in the data; the risk troubleshooting rules include one or more of low-performance data troubleshooting rules, high-risk session troubleshooting rules, memory crowding troubleshooting rules, execution plan risk troubleshooting rules, and online inspection rules;

[0145] The governance unit 1103 is used to govern the risk data and obtain a governance result.

[0146] In a possible implementation, when the risk screening rule includes a low-performance data screening rule, the screening unit is specifically configured to:

[0147] Based on the log information, determine, from each data corresponding to the log information, first data to be checked whose execution time is greater than a first threshold and whose total processor time consumed when executing the data ranks in the top N; wherein N is an integer greater than 0;

[0148] Based on the low-performance data screening rule and the log information, the first data to be screened is screened to determine risk data among the data.

[0149] In a possible implementation, when the risk screening rule includes a high-risk session screening rule, the screening unit is specifically configured to:

[0150] Based on the log information, determining, from each data corresponding to the log information, a session whose execution duration is greater than a second threshold;

[0151] Based on the high-risk session screening rule and the log information, the session is screened to determine the risk data among the data.

[0152] In a possible implementation, when the risk screening rule includes a memory occupation screening rule, the screening unit is specifically configured to:

[0153] Based on the memory occupation troubleshooting rule and the log information, data corresponding to the log information that occupies a memory greater than a third threshold is determined as risk data.

[0154] In a possible implementation, when the risk screening rule includes an execution plan risk screening rule, the screening unit is specifically configured to:

[0155] Based on the log information, determine, from each data corresponding to the log information, second data to be checked whose execution time is greater than a fourth threshold and whose total processor time consumed when executing the data ranks in the top M positions; wherein M is an integer greater than 0;

[0156] Determining text data corresponding to the second data to be checked based on the information identifier of the second data to be checked;

[0157] Based on the execution plan risk screening rules and the log information, the text data is screened to determine the risk data in the data.

[0158] In a possible implementation, when the risk screening rule includes an online inspection rule, the screening unit is specifically configured to:

[0159] Based on the online inspection rules and the log information, a database corresponding to the log information is scanned to determine the risk data in the data.

[0160] In a possible implementation, the governance unit is specifically configured to:

[0161] Storing the risk data in a risk data list;

[0162] In response to a user's modification operation on risk data in the risk data list, obtaining governed data;

[0163] Based on the managed data, the risk data list is updated to obtain a management result.

[0164] To summarize, the embodiment of the present application provides a data governance device which, based on risk screening rules and the log information, can screen low-performance data, high-risk sessions, data with high memory usage, and risky execution plans, thereby achieving full life cycle data governance and ensuring the stability of the database.

[0165] An embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory;

[0166] The memory is used to store program code and transmit the program code to the processor;

[0167] The processor is used to execute the steps of a data governance method as described above according to the instructions in the program code.

[0168] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a data governance method as described above are implemented.

[0169] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data governance method, characterized in that: include: Based on the connection information of the database, obtaining log information in the database; Based on the risk screening rules and the log information, each data corresponding to the log information is screened to determine the risk data in the data; the risk screening rules include one or more of low-performance data screening rules, high-risk session screening rules, memory crowding screening rules, execution plan risk screening rules, and online inspection rules; The risk data is governed to obtain a governance result.

2. The method according to claim 1, characterized in that When the risk screening rule includes a low-performance data screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes: Based on the log information, determine, from each data corresponding to the log information, first data to be checked whose execution time is greater than a first threshold and whose total processor time consumed when executing the data ranks in the top N; wherein N is an integer greater than 0; Based on the low-performance data screening rule and the log information, the first data to be screened is screened to determine risk data among the data.

3. The method according to claim 1, characterized in that When the risk screening rule includes a high-risk session screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes: Based on the log information, determining, from each data corresponding to the log information, a session whose execution duration is greater than a second threshold; Based on the high-risk session screening rule and the log information, the session is screened to determine the risk data among the data.

4. The method according to claim 1, characterized in that: When the risk screening rule includes a memory occupation screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes: Based on the memory occupation troubleshooting rule and the log information, data corresponding to the log information that occupies a memory greater than a third threshold is determined as risk data.

5. The method according to claim 1, characterized in that When the risk screening rule includes an execution plan risk screening rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes: Based on the log information, determine, from each data corresponding to the log information, second data to be checked whose execution time is greater than a fourth threshold and whose total processor time consumed when executing the data ranks in the top M positions; wherein M is an integer greater than 0; Determining text data corresponding to the second data to be checked based on the information identifier of the second data to be checked; Based on the execution plan risk screening rules and the log information, the text data is screened to determine the risk data in the data.

6. The method according to claim 1, characterized in that When the risk screening rule includes an online inspection rule, screening each data corresponding to the log information based on the risk screening rule and the log information to determine the risk data in the data includes: Based on the online inspection rules and the log information, a database corresponding to the log information is scanned to determine the risk data in the data.

7. The method according to any one of claims 1 to 6, characterized in that: The step of governing the risk data and obtaining a governance result includes: Storing the risk data in a risk data list; In response to a user's modification operation on risk data in the risk data list, obtaining governed data; Based on the managed data, the risk data list is updated to obtain a management result.

8. A data management device, characterized in that: include: Acquisition unit, troubleshooting unit, and governance unit; The acquisition unit is used to acquire the log information in the database based on the connection information of the database; The troubleshooting unit is used to troubleshoot each data corresponding to the log information based on the risk troubleshooting rules and the log information, and determine the risk data in the data; the risk troubleshooting rules include one or more of low-performance data troubleshooting rules, high-risk session troubleshooting rules, memory crowding troubleshooting rules, execution plan risk troubleshooting rules, and online inspection rules; The governance unit is used to govern the risk data and obtain governance results.

9. A computer device, characterized in that: The computer device comprises: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of a data governance method as described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a data governance method as described in any one of claims 1 to 7.

Images