GoldenDB database change risk management and control method and system

Abstract

The invention discloses a GoldenDB database change risk management and control method and system. The method comprises the steps that change scheme data is obtained, a coding format is recognized, effective change content is extracted, the change content is disassembled into single transactions, semantic analysis is executed, and change types are classified based on keyword matching and a rule base; wherein the change types comprise data change, structure change, parameter change and permission change; generating a distributed execution plan according to the change types, analyzing the distributed execution plan, executing information in parallel, and determining data nodes and computing nodes corresponding to the four change types respectively; respectively constructing risk assessment models in the execution domains corresponding to the four change types; and converting the risk assessment result of each node into a standardized data format, aggregating and analyzing the cross-node common risk, generating a father-child tree management and control report, and executing blocking, rollback or alarm operation according to the risk level.

Description

Technical Field

[0001] This application relates to the technical field of database change control, and particularly to a method and system for controlling the change risks of GoldenDB databases. Background Art GoldenDB databases have been widely used in the field of information technology innovation in the financial industry. However, the related ecological construction of GoldenDB databases in the industry has been slow, and no effective solutions have been proposed for controlling the change risks of GoldenDB databases.

[0002] In the traditional control methods for changes in centralized databases, usually only aspects related to the success or failure of changes, such as SQL syntax, database permissions, and whitelists, are controlled. Insufficient attention is paid to the risks caused by database changes, or only the risks of centralized data nodes are controlled. As a typical representative of distributed databases, a single change in a GoldenDB database usually affects multiple nodes and data shards, and directly affects the capital security of customers in the financial industry. Therefore, there is currently a lack of a set of control strategies for the change risks of GoldenDB databases. Summary of the Invention

[0003] This application provides a method and system for controlling the change risks of GoldenDB databases to solve the above problems.

[0004] On the one hand, this application provides a method for controlling the change risks of GoldenDB databases. The method includes the following steps: Step S1: Obtain change plan data through an automated interface or manual input, identify the encoding format and extract the effective change content, disassemble the change into single transactions and then perform semantic parsing, and classify the change into four types of operations: data, structure, parameter, and permission based on keyword matching and a rule library; Step S2: Generate a distributed execution plan according to the change type and GoldenDB pre-execution technology, parse the shard pruning, calculation pushdown, and parallel execution information in the plan, determine the data nodes and calculation nodes corresponding to data, structure, and parameter changes, and limit permission changes to calculation nodes; Step S3: Construct four types of risk assessment models for permissions, parameters, data, and structures in the corresponding execution domains respectively; among them, the permission risk assessment model is based on the minimum permission rule and the high-risk operation blocking mechanism, the parameter risk assessment model combines the parameter dictionary with manual configuration review, the data risk assessment model conducts multi-dimensional analysis through the resource layer, data layer, service layer, and operation and maintenance layer, and the structure risk assessment model evaluates business stability and structural rationality; Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze the common risks across nodes, generate a control report in a parent-child tree form, and perform blocking, rollback, or warning operations according to the risk level.

[0005] In an implementation manner of the present application, in step S3, the specific implementation of the permission risk assessment model includes: parsing the changed user permission scope, verifying whether it conforms to the least privilege rule in combination with the distributed execution plan, identifying high-risk permission operations and blocking them; wherein, the high-risk permissions refer to the permissions that affect the database cluster configuration level, such as granting a user the permission to modify the cluster sql mode. This authorization operation will be blocked at the system level and will not enter the database.

[0006] In an implementation manner of the present application, the specific implementation of the parameter risk assessment model includes: extracting the parameter dictionaries of the computing nodes and data nodes according to the execution domain, matching the parameter types involved in the change, and generating a parameter impact report; for high-risk parameter changes, operation and maintenance personnel need to manually configure audit rules, and the changes that do not pass the audit are directly blocked, and the historical audit results are recorded through the control report. Among them, high-risk parameters are equivalent to the parameters with greater common impact divided in the parameter dictionary (such as the return timeout of the data node). Because different systems have different perceptions of high risk (for example, System A processes online transaction services, sets a 5-minute return timeout, and the database does not return data after reaching the threshold, then an error will occur at the system level, which will affect the business. However, System B processes batch processing and has a lower requirement for data return time). Therefore, operation and maintenance personnel need to manually configure audit rules for different high-risk parameters according to the business scenario.

[0007] In an implementation manner of the present application, the specific implementation of the data risk assessment model includes: judging whether the change causes resource competition, replication delay, data skew and service interruption by analyzing the resource layer load, data layer synchronization status, service layer throughput and operation and maintenance layer health indicators of the data node; if a data risk point is detected, a high-risk alarm is generated; wherein, the data risk points include but are not limited to: the node disk I / O exceeds the threshold, the transaction conflict rate is abnormal, and the master-slave delay exceeds the limit; if the database cluster status is detected to be abnormal during the execution of the change, the change rollback is triggered and the resources are released. In an implementation manner of the present application, the specific implementation of the structure risk assessment model includes: identifying non-online operations and hot tables in the change, and evaluating the impact of the structure change on business stability; checking hot data, index coverage, primary key constraints, partition expansion, and data expansion risk in combination with the table structure rationality rules; if a structure risk point is detected, a high-risk alarm is generated; wherein, the structure risk points include but are not limited to: the hot table does not enable read-write separation and the deletion of the hot index causes the business to slow down.

[0008] In an implementation manner of the present application, in step S4, the aggregation analysis includes: classifying the risk points of the same change in different shards by type, counting the common risk indicators, predicting the trend of cluster-level load imbalance or data skew, and displaying the change impact path in the form of a parent-child tree to locate high-risk nodes and associated shards.

[0009] In an implementation manner of the present application, the pre-execution technology specifically includes: performing distributed optimization on the change content through the GoldenDB computing nodes, including statement rewriting, shard pruning, and parallel execution planning, generating an execution plan containing the data shard mapping relationship as the basis for execution domain determination and risk assessment.

[0010] In an implementation manner of the present application, in step S2, during the change parsing process, a rule library is used to integrate the data node parameter rules, computing node parameter rules, and middleware parameter rules, extract keywords through semantic parsing, and dynamically update the rule library to adapt to the GoldenDB version upgrade.

[0011] The present application also provides a control system for the change risk of the GoldenDB database. The system includes: a change collection and parsing module for obtaining and parsing a change plan and outputting a single transaction operation after classification; an execution domain analysis module for determining the nodes and shard ranges affected by the change based on the pre-execution technology; a risk assessment module including four sub-models of permission, parameter, data, and structure for performing multi-dimensional risk analysis on the computing nodes and data nodes respectively; and a risk summary module for aggregating cross-node risk data, generating a control report, and triggering a control instruction.

[0012] In an implementation manner of the present application, the risk assessment module further includes: a real-time monitoring unit for collecting node resource status, transaction conflict rate, and cluster health indicators; a dynamic blocking unit for automatically intercepting high-risk changes according to the risk assessment results and notifying the operation and maintenance personnel synchronously; and a change backtracking unit for recording all change operations, risk assessment processes, and control results, supporting historical data query and auditing.

[0013] The control method and system for the change risk of the GoldenDB database provided by the present application have the following beneficial effects: (1) By means of automated control and risk assessment, the risks such as inconsistent shard data in the database, decreased database performance, or business interruption caused by changes are significantly reduced; (2) Based on the parsing results of the GoldenDB database change plan information, combined with the current state of the database, conduct a change risk analysis, issue the execution plan generated through change parsing, determine the data shards affected by the change, and based on the execution plan in a single shard, determine the risk impacts of the change in this shard. Finally, summarize the risk results in all data shards and generate a risk control report aiming at business stability. (3) The operation and maintenance personnel can, based on the risk control report, more quickly understand the risks existing in the current change in the GoldenDB database, thereby better ensuring the reliability and stability of the relevant business systems. Description of the Drawings

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings: Figure 1 It is a flowchart of a method for controlling the change risk of a GoldenDB database provided by an embodiment of the present application; Figure 2 It is a composition diagram of a system for controlling the change risk of a GoldenDB database provided by an embodiment of the present application. Detailed Embodiments

[0015] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0016] The embodiments of the present application provide a method and system for controlling the change risk of a GoldenDB database. The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the drawings.

[0017] Figure 1 It is a flowchart of a method for controlling the change risk of a GoldenDB database provided by an embodiment of the present application. As Figure 1 shown, the method mainly includes the following steps: Step S1: Obtain the change plan data, identify the encoding format, extract the effective change content, disassemble the change content into single transactions, and then perform semantic parsing. Classify the change types based on keyword matching and the rule library; wherein, the change types include: data change, structure change, parameter change, and permission change.

[0018] In the embodiment of the present application, the sources of the change plan mainly include two parts: one is to obtain the change file through the change platform and provide it through FTP file transfer; the other is that the operation and maintenance personnel manually input the change plan. The system will automatically identify the character encoding of the change file and extract the valid change content data in the file; for the manually input change content, the system front end will review it to ensure the validity of the data.

[0019] The change parsing process includes: splitting the acquired change data into changes of a single database transaction, and performing semantic parsing on the split changes according to the parsing rules. The parsing rules integrate DN data node parameter rules, CN computing node parameter rules, database middleware (mainly CN and DN connection middleware, such as GTM) parameter rules, etc. Semantic parsing extracts lexical information from a single change, matches keywords (such as SET, DROP, CREATE, INSERT, GRANT, etc., SET at the beginning indicates parameter changes, DROP and CREATE are usually table structure changes, INSERT is usually data changes, and GRANT is usually permission changes.), and combines the parsing rules to finally classify the change plan into four types of change operations: data, structure, parameters, and permissions.

[0020] Step S2: Generate a distributed execution plan according to the change type, parse the distributed execution plan, execute information in parallel, and determine the data nodes and computing nodes corresponding to the four change types.

[0021] In the embodiment of the present application, the execution domain of permission changes only exists in the CN computing node. After the system is connected to the CN computing node, subsequent model evaluation, risk aggregation and other tasks are completed in the CN computing node. The execution domain of data changes, structural changes, and parameter changes is determined by GoldenDB's pre-execution technology. Pre-execution technology is the optimization of the change content by the GoldenDB computing node in the distributed cluster based on the cluster information, including content rewriting, sharding, calculation pushdown, parallel execution, etc., and generates a distributed execution plan.

[0022] The system extracts data shards, distributed execution statements, and parallel execution information from the change execution plan, determines the shard pruning and calculation pushdown of the change, and then understands whether the change is rewritten and sent, the sent data shards, and the rewritten execution plan. Usually, data and structural changes will be rewritten by the computing nodes and then sent to the data nodes, while the sending of parameter changes is specified by the change plan, and the computing nodes determine whether to execute based on the cluster status.

[0023] Step S3: Build four types of risk assessment models for data, structure, parameters, and permissions in the corresponding execution domains of the four change types; among them, the permission risk assessment model is based on the least privilege rule and the high-risk operation blocking mechanism, the parameter risk assessment model combines the parameter dictionary and manual configuration review, the data risk assessment model conducts multi-dimensional analysis through the resource layer, data layer, service layer, and operation and maintenance layer, and the structure risk assessment model evaluates business stability and structural rationality.

[0024] In the embodiment of the present application, the permission assessment model is only built on the CN computing node of GoldenDB to conduct risk assessment for permission change operations. The model analyzes the permission scope of the changed user, and combines the change execution plan to evaluate whether it conforms to the least privilege rule (for example, the gdblook user can only have the select permission), and identifies over-privilege operations (such as permissions that cannot be granted to users). For high-risk permission operations (such as granting users the permission to modify the cluster SQL mode), the system directly blocks and generates a risk warning.

[0025] The parameter assessment model is built on the computing node or data node of GoldenDB to conduct risk assessment for parameter change operations. The model generates a parameter dictionary by analyzing the parameters involved in the CN computing node and the DN data node. Combining the change execution domain and the parameter content, the system generates a parameter impact report. For high-risk parameter changes, the operation and maintenance personnel need to manually configure the review rules, and the changes that do not pass the review will be blocked.

[0026] The data assessment model is built on the data node of GoldenDB to conduct risk assessment for data change operations. The model analyzes from four dimensions: the resource layer, the data layer, the service layer, and the operation and maintenance layer: (1) Resource layer: Analyze the CPU usage rate, network bandwidth, memory usage rate, and disk I / O of the node to determine whether the change will cause resource competition or block other threads. (2) Data layer: Analyze the data synchronization status and the cluster data distribution, and combine the size of the transaction data volume to determine whether the change will cause master-slave replication delay or data skew. (3) Service layer: Analyze the node throughput (QPS / TPS) and the number of connections to determine whether the change will be affected by the load bottleneck. (4) Operation and maintenance layer: Monitor the health status of the cluster (such as shard health, backup situation, log inflation), and block the change and roll back the operation when an abnormality is found.

[0027] The structure evaluation model is built on the data nodes of GoldenDB to conduct risk assessment for structure change operations. The model includes business stability assessment and structural rationality assessment: (1) Business stability assessment: Check non-online operations (such as blocking operations) and hot tables in the change to determine whether it will cause resource competition or business interruption. (2) Structural rationality assessment: Check the rationality of table structure changes, including modification of data column types, addition and deletion of indexes (to avoid data expansion or deletion of hot indexes), primary key constraints, partition expansion, and data expansion risks.

[0028] Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze cross-node common risks, generate a parent-child tree-style control report, and perform blocking, rollback, or warning operations according to the risk level.

[0029] In the embodiment of the present application, the risk control system integrates and summarizes the risk assessment results of each node to generate a unified control report: (1) Generate standardized data: Uniformly convert the risk assessment reports of each node into JSON string format. (2) Integrate change risks: Summarize the impacts of the same change on different shards, and aggregate and analyze common risks (such as cross-node data skew). (3) Parent-child tree control report: Decompose the aggregated impact of the change and its impact relationship on different data nodes into a parent-child tree structure to help operation and maintenance personnel locate and analyze high-risk nodes and associated shards.

[0030] The above is a control system for GoldenDB database change risks provided by the embodiment of the present application. Based on the same inventive concept, the embodiment of the present application also provides a control system for GoldenDB database change risks. Figure 2 For the composition diagram of a control system for GoldenDB database change risks provided by the embodiment of the present application, as Figure 2 shown, the system mainly includes: a change collection and parsing module 201, which is used to obtain and parse the change plan and output single transaction operations after classification; an execution domain analysis module 202, which determines the nodes and shard ranges affected by the change based on pre-execution technology; a risk assessment module 203, which includes four sub-models of permissions, parameters, data, and structure, and is used to conduct multi-dimensional risk analysis for the compute nodes and data nodes respectively; a risk summary module 204, which is used to aggregate cross-node risk data, generate a control report, and trigger a control instruction.

[0031] In the embodiment of the present application, the risk assessment module 203 further includes: a real-time monitoring unit, which is used to collect node resource status, transaction conflict rate, and cluster health indicators; a dynamic blocking unit, which is used to automatically intercept high-risk changes according to the risk assessment results and synchronously notify operation and maintenance personnel; a change backtracking unit, which is used to record all change operations, risk assessment processes, and control results, and support historical data query and auditing.

[0032] This system has been launched in the pre-production environment, providing services for more than 140 systems. The following is an example of the control report generated by the system: Cluster name: DB-Cluster-01, number of nodes: 6, number of shards: 3, number of replicas: 2.

[0033] Change 1201512: The session_lock_timeout parameter is at the session level and only affects the lock timeout in the current session.

[0034] Change 1201513: Involves DN3 and DN5, operates on the hot table user table, and deleting partitions is likely to cause contention for lock resources.

[0035] - Data node 3: The user table has been accessed multiple times within 30 seconds on the current node. There is no read-write separation in the distributed execution plan.

[0036] - Data node 5: The user table has been accessed multiple times within 30 seconds on the current node. There is no read-write separation in the distributed execution plan.

[0037] Change 1201514: The number of updated rows in a single transaction exceeds 10,000, which is likely to cause master-slave replication delay and data skew. It is recommended to split the change.

[0038] - Data node 3: The number of transaction changes on the current node is 7634 rows, and exceeding 4000 rows in a single transaction is likely to cause node replication delay.

[0039] - Data node 5: The number of transaction changes on the current node is 4437 rows, and exceeding 4000 rows in a single transaction is likely to cause node replication delay.

[0040] - Data node 5: The current user table accounts for more than 10% on this node, and the excessive transaction volume is likely to cause data skew.

[0041] Change 1201515: The disk I / O of DN5 exceeds 80%, which is likely to cause transaction commit delay.

[0042] - Data node 5: The disk IO of the current node exceeds 80%, and transactions on this node are likely to have commit delay.

[0043] Change 1201516: It is a privilege escalation operation, and the ods user has no right to modify the drop permission of the user table.

[0044] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the description of the method embodiments.

[0045] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.

[0046] The above description is only for the embodiments of this application and is not intended to limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims

1. A method for controlling the change risk of GoldenDB database, characterized in that, The method includes the following steps: Step S1: Obtain change plan data, identify the encoding format, extract the effective change content, disassemble the change content into single transactions, perform semantic parsing, and classify the change types based on keyword matching and a rule library; wherein, the change types include: data change, structure change, parameter change, and permission change; Step S2: Generate a distributed execution plan according to the change type, parse the distributed execution plan, execute the information in parallel, and determine the data nodes and computing nodes corresponding to the four change types respectively; Step S3: Build four types of risk assessment models for data, structure, parameters, and permissions in the execution domains corresponding to the four change types respectively; wherein, the permission risk assessment model is based on the least privilege rule and the high-risk operation blocking mechanism, the parameter risk assessment model combines the parameter dictionary with manual configuration review, the data risk assessment model conducts multi-dimensional analysis through the resource layer, data layer, service layer, and operation and maintenance layer, and the structure risk assessment model evaluates the business stability and structural rationality; Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze the common risks across nodes, generate a parent-child tree-style control report, and perform blocking, rollback, or warning operations according to the risk level.

2. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, In the said Step S3, the specific implementation of the permission risk assessment model includes: Parse the permission scope of the changed user, verify whether it conforms to the least privilege rule in combination with the distributed execution plan, identify high-risk permission operations and block them; wherein, the high-risk permissions are the permissions that affect the database cluster configuration level.

3. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, The specific implementation of the parameter risk assessment model includes: Extract the parameter dictionaries of the computing nodes and data nodes according to the execution domain, match the parameter types involved in the change, and generate a parameter impact report; For high-risk parameter changes, receive the review rules configured by the operation and maintenance personnel, block the changes that do not pass the review, and record the historical review results through the control report.

4. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, The specific implementation of the data risk assessment model includes: Analyze the resource layer load, data layer synchronization status, service layer throughput, and operation and maintenance layer health indicators of the data nodes, and determine whether the change causes resource competition, replication delay, data skew, and service interruption; If a data risk point is detected, generate a high-risk alarm; wherein, the data risk points include but are not limited to: the node disk I / O exceeds the threshold, the transaction conflict rate is abnormal, and the master-slave delay exceeds the limit; If the database cluster status is monitored to be abnormal during the change execution process, trigger a change rollback and release the resources.

5. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, The specific implementation of the structure risk assessment model includes: Identify the non-online operations and hot tables in the change, and evaluate the impact of the structure change on business stability; Check the hot data, index coverage, primary key constraint, partition expansion, and data expansion risk based on the table structure rationality rules; If a structure risk point is detected, generate a high-risk alarm; wherein, the structure risk points include but are not limited to: the hot table does not enable read-write separation and the hot index is deleted, causing the business to slow down.

6. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, In the said Step S4, the aggregation analysis includes: Classify the risk points of the same change in different shards by type, count the common risk indicators, predict the trend of cluster-level load imbalance or data skew, and display the change impact path in the form of a parent-child tree to locate high-risk nodes and associated shards.

7. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, Generate a distributed execution plan according to the change type, including generating a distributed execution plan based on the pre-execution technology of the GoldenDB database. The specific pre-execution technology includes: Perform distributed optimization on the change content through GoldenDB computing nodes, including statement rewriting, shard pruning, and parallel execution planning, and generate an execution plan containing data shard mapping relationships as the basis for execution domain determination and risk assessment.

8. The method for controlling the change risk of GoldenDB database according to claim 1, characterized in that, In step S2, during the change parsing process, integrate the data node parameter rules, computing node parameter rules, and middleware parameter rules using a rule library, extract keywords through semantic parsing, and dynamically update the rule library to adapt to the GoldenDB version upgrade.

9. A control system for the change risk of GoldenDB database, characterized in that, The system includes: A change collection and parsing module for obtaining and parsing a change plan and outputting single transaction operations after classification; An execution domain analysis module for determining the nodes and shard ranges affected by the change based on pre-execution technology; A risk assessment module, including four sub-modules of permissions, parameters, data, and structure, for performing multi-dimensional risk analysis on computing nodes and data nodes respectively; A risk summary module for aggregating cross-node risk data, generating a control report, and triggering a control instruction.

10. A control system for the risk of GoldenDB database changes according to claim 9, characterized in that The risk assessment module further includes: A real-time monitoring unit for collecting node resource status, transaction conflict rate, and cluster health indicators; A dynamic blocking unit for automatically intercepting high-risk changes according to the risk assessment results and synchronously notifying the operation and maintenance personnel; A change backtracking unit for recording all change operations, risk assessment processes, and control results, and supporting historical data query and auditing.

Images