A Method and System for Controlling the Risks of GoldenDB Database Changes

Abstract

The present application discloses a control method and system for the change risks of the GoldenDB database, including: obtaining change plan data and identifying the encoding format, extracting valid change content, disassembling the change content into single transactions and then performing semantic parsing, and classifying the change types based on keyword matching and a rule library; wherein, the change types include: data change, structure change, parameter change, and permission change; generating a distributed execution plan according to the change types, parsing the distributed execution plan, parallelly executing information, and determining the data nodes and computing nodes corresponding to the four change types respectively; constructing risk assessment models respectively in the execution domains corresponding to the four change types; converting the risk assessment results of each node into a standardized data format, aggregating and analyzing the common risks across nodes, generating a parent-child tree control report, and performing blocking, rollback, or warning operations according to the risk level.

Description

Technical Field

[0001] This application relates to the technical field of database change control, and particularly to a method and system for controlling the change risks of GoldenDB databases. Background Art

[0002] GoldenDB databases have been widely used in the information technology innovation field of the financial industry. However, the related ecological construction of GoldenDB databases in the industry is slow, and no effective solutions have been proposed for controlling the change risks of GoldenDB databases.

[0003] In the traditional control methods for changes in centralized databases, usually only aspects related to the success or failure of changes, such as SQL syntax, database permissions, and whitelists, are controlled. Insufficient attention is paid to the risks caused by database changes, or only the risks of centralized data nodes are controlled. As a typical representative of distributed databases, a single change in a GoldenDB database usually affects multiple nodes and data shards, and directly affects the capital security of customers in the financial industry. Therefore, there is currently a lack of a set of control strategies for the change risks of GoldenDB databases. Summary of the Invention

[0004] This application provides a method and system for controlling the change risks of GoldenDB databases to solve the above problems.

[0005] On the one hand, this application provides a method for controlling the change risks of GoldenDB databases. The method includes the following steps: Step S1: Obtain change plan data through an automated interface or manual input, identify the encoding format and extract the effective change content, disassemble the change into single transactions and then perform semantic parsing, and classify the change into four types of operations: data, structure, parameter, and permission based on keyword matching and a rule library; Step S2: Generate a distributed execution plan according to the change type and GoldenDB pre-execution technology, parse the shard pruning, calculation pushdown, and parallel execution information in the plan, determine the data nodes and calculation nodes corresponding to data, structure, and parameter changes, and limit permission changes to calculation nodes; Step S3: Construct four types of risk assessment models for permissions, parameters, data, and structures in the corresponding execution domains respectively; among them, the permission risk assessment model is based on the minimum permission rule and the high-risk operation blocking mechanism, the parameter risk assessment model combines the parameter dictionary with manual configuration review, the data risk assessment model performs multi-dimensional analysis through the resource layer, data layer, service layer, and operation and maintenance layer, and the structure risk assessment model evaluates business stability and structural rationality; Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze the common risks across nodes, generate a control report in the form of a parent-child tree, and perform blocking, rollback, or warning operations according to the risk level.

[0006] In an implementation of the present application, in step S3, the specific implementation of the permission risk assessment model includes: parsing the changed user permission scope, verifying whether it conforms to the least privilege rule in combination with the distributed execution plan, identifying high-risk permission operations and blocking them; where the high-risk permissions refer to the permissions that affect the database cluster configuration level, such as granting a user the permission to modify the cluster sql mode. This authorization operation will be blocked at the system level and will not enter the database.

[0007] In an implementation of the present application, the specific implementation of the parameter risk assessment model includes: extracting the parameter dictionaries of the computing nodes and data nodes according to the execution domain, matching the parameter types involved in the change, and generating a parameter impact report; for high-risk parameter changes, the operation and maintenance personnel need to manually configure the audit rules, and the changes that do not pass the audit are directly blocked, and the historical audit results are recorded through the control report. Among them, the high-risk parameters are equivalent to the parameters with relatively large common impacts divided in the parameter dictionary (such as the return timeout time of the data node). Because different systems have different sensitivities to high risks (for example, System A processes online transaction services, sets a 5-minute return timeout time, and when the threshold is reached, the database does not return data, then an error will occur at the system level, thus affecting the business. However, System B processes batch processing and has a lower requirement for data return time). Therefore, the operation and maintenance personnel need to manually configure the audit rules for different high-risk parameters according to the business scenario.

[0008] In an implementation of the present application, the specific implementation of the data risk assessment model includes: judging whether the change causes resource competition, replication delay, data skew, and service interruption by analyzing the resource layer load, data layer synchronization status, service layer throughput, and operation and maintenance layer health indicators of the data nodes; if a data risk point is detected, a high-risk alarm is generated; where the data risk points include but are not limited to: the node disk I / O exceeds the threshold, the transaction conflict rate is abnormal, and the master-slave delay exceeds the limit; if the database cluster status is detected to be abnormal during the execution of the change, the change rollback is triggered and the resources are released. In an implementation of the present application, the specific implementation of the structure risk assessment model includes: identifying the non-online operations and hot tables in the change, and evaluating the impact of the structure change on the business stability; checking the hot data, index coverage, primary key constraint, partition expansion, and data expansion risk in combination with the table structure rationality rules; if a structure risk point is detected, a high-risk alarm is generated; where the structure risk points include but are not limited to: the hot table does not enable read-write separation and the deletion of the hot index causes the business to slow down.

[0009] In an implementation manner of the present application, in step S4, the aggregation analysis includes: classifying the risk points of the same change in different shards by type, counting the common risk indicators, predicting the trend of load imbalance or data skew at the cluster level, and displaying the change impact path in the form of a parent-child tree to locate high-risk nodes and associated shards.

[0010] In an implementation manner of the present application, the pre-execution technology specifically includes: performing distributed optimization on the change content through the GoldenDB computing nodes, including statement rewriting, shard pruning, and parallel execution planning, generating an execution plan containing data shard mapping relationships as the basis for execution domain determination and risk assessment.

[0011] In an implementation manner of the present application, in step S2, during the change parsing process, a rule library is used to integrate the data node parameter rules, computing node parameter rules, and middleware parameter rules, keywords are extracted through semantic parsing, and the rule library is dynamically updated to adapt to the GoldenDB version upgrade.

[0012] The present application also provides a control system for the change risk of the GoldenDB database. The system includes: a change collection and parsing module for obtaining and parsing a change plan and outputting a single transaction operation after classification; an execution domain analysis module for determining the nodes and shard ranges affected by the change based on the pre-execution technology; a risk assessment module including four sub-models of permission, parameter, data, and structure for performing multi-dimensional risk analysis on the computing nodes and data nodes respectively; and a risk summary module for aggregating cross-node risk data, generating a control report, and triggering a control instruction.

[0013] In an implementation manner of the present application, the risk assessment module further includes: a real-time monitoring unit for collecting node resource status, transaction conflict rate, and cluster health indicators; a dynamic blocking unit for automatically intercepting high-risk changes according to the risk assessment results and synchronously notifying the operation and maintenance personnel; and a change backtracking unit for recording all change operations, risk assessment processes, and control results, supporting historical data query and auditing.

[0014] The control method and system for the change risk of the GoldenDB database provided by the present application have the following beneficial effects:

[0015] (1) By means of automated control and risk assessment, the risks such as inconsistent shard data of the database, decreased database performance, or business interruption caused by changes are significantly reduced.

[0016] (2) Based on the parsing results of the GoldenDB database change plan information, combined with the current state of the database, conduct a change risk analysis, issue the execution plan generated through change parsing, determine the data shards affected by the change, and based on the execution plan in a single shard, determine the risk impacts of the change in this shard. Finally, summarize the risk results in all data shards and generate a risk control report aiming at business stability.

[0017] (3) The operation and maintenance personnel can, according to the risk control report, more quickly understand the risks existing in the current change in the GoldenDB database, thereby better ensuring the reliability and stability of the relevant business systems. Description of the Drawings

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0019] Figure 1 It is a flowchart of a method for controlling the change risk of a GoldenDB database provided by an embodiment of the present application;

[0020] Figure 2 It is a composition diagram of a system for controlling the change risk of a GoldenDB database provided by an embodiment of the present application. Detailed Embodiments

[0021] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0022] The embodiments of the present application provide a method and a system for controlling the change risk of a GoldenDB database. The technical solutions proposed by the embodiments of the present application will be described in detail below through the drawings.

[0023] Figure 1 It is a flowchart of a method for controlling the change risk of a GoldenDB database provided by an embodiment of the present application. As Figure 1 shown, the method mainly includes the following steps:

[0024] Step S1: Obtain change plan data, identify the encoding format, extract the effective change content, disassemble the change content into single transactions and then perform semantic parsing, and classify the change types based on keyword matching and the rule library; among them, the change types include: data change, structure change, parameter change, and permission change.

[0025] In the embodiment of the present application, the sources of the change plan mainly have two parts: one is to obtain the change file through the change platform and provide it using FTP file transfer; the other is that the operation and maintenance personnel manually input the change plan. The system will automatically identify the character encoding of the change file and extract the effective change content data in the file; for the manually input change content, the front end of the system will conduct a review to ensure the data validity.

[0026] The change parsing process includes: disassembling the obtained change data into changes of single database transactions, and performing semantic parsing on the disassembled changes according to the parsing rules. The parsing rules integrate DN data node parameter rules, CN computing node parameter rules, database middleware (mainly referring to the connection middleware between CN and DN, such as GTM) parameter rules, etc. Semantic parsing extracts lexical items from a single change, matches keywords (such as SET, DROP, CREATE, INSERT, GRANT, etc., SET at the beginning indicates parameter change, DROP and CREATE are usually table structure changes, INSERT is usually data change, and GRANT is usually permission change), and finally classifies the change plan into four types of change operations: data, structure, parameter, and permission in combination with the parsing rules.

[0027] Step S2: Generate a distributed execution plan according to the change type, parse the distributed execution plan, parallelize the execution information, and determine the data nodes and computing nodes corresponding to the four change types respectively.

[0028] In the embodiment of the present application, the execution domain of permission change only exists in the CN computing node. After the system connects to the CN computing node, subsequent model evaluation, risk aggregation, etc. are all completed in the CN computing node. The execution domains of data change, structure change, and parameter change are determined through the pre-execution technology of GoldenDB. The pre-execution technology is the optimization of the change content by the GoldenDB computing node in the distributed cluster according to the cluster information, including content rewriting, shard pruning, computation pushdown, parallel execution, etc., and generates a distributed execution plan.

[0029] The system extracts data sharding, distributed execution statements, and parallel execution information from the change execution plan to determine the sharding and computation pushdown for the change. This information then determines whether the change was rewritten and then delivered, the data shards delivered, and the rewritten execution plan. Typically, data and structural changes are rewritten by compute nodes before being delivered to data nodes. Parameter changes are delivered based on the change plan, and compute nodes determine whether to execute them based on cluster status.

[0030] Step S3: Construct four risk assessment models for data, structure, parameters, and permissions in the execution domains corresponding to the four change types. The permission risk assessment model is based on the least privilege rule and high-risk operation blocking mechanism. The parameter risk assessment model combines the parameter dictionary with manual configuration review. The data risk assessment model uses multi-dimensional analysis of the resource layer, data layer, service layer, and operation and maintenance layer. The structure risk assessment model evaluates business stability and structural rationality.

[0031] In this embodiment of the application, the permission assessment model is built only on GoldenDB's CN computing nodes to perform risk assessments on permission change operations. The model analyzes the scope of the changed user's permissions and, combined with the change execution plan, assesses compliance with the least privilege rule (e.g., the gdblook user can only have select permissions). It also identifies over-privileged operations (e.g., permissions that the user cannot grant). For high-risk permission operations (e.g., granting a user permission to modify the cluster SQL mode), the system directly blocks them and generates a risk alert.

[0032] The parameter assessment model is built on GoldenDB's compute nodes or data nodes to assess the risks of parameter changes. The model analyzes the parameters involved in CN compute nodes and DN data nodes to generate a parameter dictionary. Based on the change execution domain and parameter content, the system generates a parameter impact report. For high-risk parameter changes, operations and maintenance personnel must manually configure review rules; changes that fail review will be blocked.

[0033] The data assessment model is built on GoldenDB's data nodes to perform risk assessments on data change operations. The model analyzes data from four dimensions: resource layer, data layer, service layer, and operation and maintenance layer: (1) Resource layer: Analyzes node CPU usage, network bandwidth, memory usage, and disk I / O to determine whether the change will cause resource competition or block other threads. (2) Data layer: Analyzes data synchronization status and cluster data distribution, and combines the transaction data volume to determine whether the change will cause master-slave replication delays or data skew. (3) Service layer: Analyzes node throughput (QPS / TPS) and number of connections to determine whether the change will be affected by load bottlenecks. (4) Operation and maintenance layer: Monitors cluster health (such as shard health, backup status, and log expansion), and blocks changes and rolls back operations when anomalies are found.

[0034] The structure evaluation model is built on the data nodes of GoldenDB to conduct risk assessment for structure change operations. The model includes business stability assessment and structural rationality assessment: (1) Business stability assessment: Check non-online operations (such as blocking operations) and hot tables in the change to determine whether resource competition or business interruption will be caused. (2) Structural rationality assessment: Check the rationality of table structure changes, including modification of data column types, addition and deletion of indexes (to avoid data expansion or deletion of hot indexes), primary key constraints, partition expansion, and data expansion risks.

[0035] Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze cross-node common risks, generate a control report in a parent-child tree format, and perform blocking, rollback, or warning operations according to the risk level.

[0036] In the embodiment of the present application, the risk control system integrates and summarizes the risk assessment results of each node to generate a unified control report: (1) Generate standardized data: Uniformly convert the risk assessment reports of each node into JSON string format. (2) Integrate change risks: Summarize the impacts of the same change on different shards, and aggregate and analyze common risks (such as data skew across nodes). (3) Parent-child tree control report: Decompose the aggregated impact of the change and its impact relationship on different data nodes into a parent-child tree structure to help operation and maintenance personnel locate and analyze high-risk nodes and associated shards.

[0037] The above is a control system for GoldenDB database change risks provided by the embodiment of the present application. Based on the same inventive concept, the embodiment of the present application also provides a control system for GoldenDB database change risks. Figure 2 For the composition diagram of a control system for GoldenDB database change risks provided by the embodiment of the present application, as Figure 2 shown, the system mainly includes: a change collection and parsing module 201, which is used to obtain and parse the change plan and output single transaction operations after classification; an execution domain analysis module 202, which determines the nodes and shard ranges affected by the change based on pre-execution technology; a risk assessment module 203, which includes four sub-models of permission, parameter, data, and structure, and is used to perform multi-dimensional risk analysis on the computing nodes and data nodes respectively; a risk summary module 204, which is used to aggregate cross-node risk data, generate a control report, and trigger a control instruction.

[0038] In the embodiment of the present application, the risk assessment module 203 further includes: a real-time monitoring unit, which is used to collect node resource status, transaction conflict rate, and cluster health indicators; a dynamic blocking unit, which is used to automatically intercept high-risk changes according to the risk assessment results and synchronously notify operation and maintenance personnel; a change backtracking unit, which is used to record all change operations, risk assessment processes, and control results, and support historical data query and auditing.

[0039] This system has been launched in the pre-production environment and provides services for more than 140 systems. The following is an example of the control report generated by the system:

[0040] Cluster name: DB-Cluster-01, number of nodes: 6, number of shards: 3, number of replicas: 2.

[0041] Change 1201512: The session_lock_timeout parameter is at the session level and only affects the lock timeout in the current session.

[0042] Change 1201513: Involves DN3 and DN5, operates on the hot table user table, and deleting partitions is likely to cause contention for lock resources.

[0043] - Data node 3: The user table is accessed multiple times within 30 seconds in the current node. There is no read-write separation in the distributed execution plan.

[0044] - Data node 5: The user table is accessed multiple times within 30 seconds in the current node. There is no read-write separation in the distributed execution plan.

[0045] Change 1201514: When the number of rows updated in a single transaction exceeds 10,000, it is likely to cause master-slave replication delay and data skew. It is recommended to split the change.

[0046] - Data node 3: The transaction change volume in the current node is 7,634 rows, and a single transaction exceeding 4,000 rows is likely to cause node replication delay.

[0047] - Data node 5: The transaction change volume in the current node is 4,437 rows, and a single transaction exceeding 4,000 rows is likely to cause node replication delay.

[0048] - Data node 5: The current user table accounts for more than 10% in this node, and the large transaction volume is likely to cause data skew.

[0049] Change 1201515: The disk I / O of DN5 exceeds 80%, which is likely to cause transaction commit delay.

[0050] - Data node 5: The disk IO of the current node exceeds 80%, and transactions in this node are likely to have commit delay.

[0051] Change 1201516: It is a privilege escalation operation, and the ods user has no right to modify the drop permission of the user table.

[0052] The embodiments in the present application are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and for the relevant parts, reference can be made to the corresponding descriptions in the method embodiments.

[0053] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.

[0054] The above are only the embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A control method for the change risks of the GoldenDB database, characterized in that The method includes the following steps: Step S1: Obtain change plan data, identify the encoding format, extract the effective change content, disassemble the change content into single transactions, perform semantic parsing, and classify the change types based on keyword matching and a rule library; wherein, the change types include: data change, structure change, parameter change, and permission change; Step S2: Generate a distributed execution plan according to the change type, parse the distributed execution plan, parallelize the execution information, and determine the data nodes and computing nodes corresponding to the four change types respectively; generating the distributed execution plan according to the change type includes generating a distributed execution plan based on the pre-execution technology of the GoldenDB database, and the pre-execution technology specifically includes: performing distributed optimization on the change content through the GoldenDB computing node, including statement rewriting, shard pruning, and parallel execution planning, generating an execution plan including data shard mapping relationships, and using it as the basis for execution domain determination and risk assessment; Step S3: Build four types of risk assessment models for data, structure, parameters, and permissions in the execution domains corresponding to the four change types respectively; among them, the permission risk assessment model is based on the least privilege rule and the high-risk operation blocking mechanism, the parameter risk assessment model combines the parameter dictionary and manual configuration review, the data risk assessment model performs multi-dimensional analysis through the resource layer, data layer, service layer, and operation and maintenance layer, and the structure risk assessment model evaluates business stability and structural rationality; Step S4: Convert the risk assessment results of each node into a standardized data format, aggregate and analyze the common risks across nodes, generate a parent-child tree-style control report, and perform blocking, rollback, or warning operations according to the risk level.

2. The control method for the change risk of the GoldenDB database according to claim 1, wherein In the above Step S3, the specific implementation of the permission risk assessment model includes: Parse the change user's permission scope, verify whether it conforms to the least privilege rule in combination with the distributed execution plan, identify high-risk permission operations and block them; wherein, the high-risk permissions are the permissions that affect the database cluster configuration level.

3. The control method for the change risk of the GoldenDB database according to claim 1, wherein The specific implementation of the parameter risk assessment model includes: Extract the parameter dictionaries of the computing node and the data node according to the execution domain, match the parameter types involved in the change, and generate a parameter impact report; For high-risk parameter changes, receive the review rules configured by the operation and maintenance personnel, block the changes that do not pass the review, and record the historical review results through the control report.

4. The control method for the change risk of the GoldenDB database according to claim 1, wherein The specific implementation of the data risk assessment model includes: Analyze the resource layer load, data layer synchronization status, service layer throughput, and operation and maintenance layer health indicators of the data node, and judge whether the change causes resource competition, replication delay, data skew, and service interruption; If a data risk point is detected, generate a high-risk alarm; wherein, the data risk points include but are not limited to: node disk I / O exceeding the threshold, transaction conflict rate anomaly, master-slave delay exceeding the limit; If the database cluster status is abnormal during the change execution process, trigger a change rollback and release the resources.

5. The control method for the change risk of the GoldenDB database according to claim 1, wherein, The specific implementation of the structure risk assessment model includes: Identify non-online operations and hot tables in the change, and evaluate the impact of the structure change on business stability; Check hot data, index coverage, primary key constraints, partition expansion, and data expansion risks based on the rationality rules of the table structure; If a structural risk point is detected, a high-risk warning is generated; among them, the structural risk points include but are not limited to: the hot table does not enable read-write separation and the hot index is deleted, resulting in slower business operations.

6. The control method for the change risk of a GoldenDB database according to claim 1, characterized in that, In step S4, the aggregation analysis includes: Classify the risk points of the same change in different shards by type, count the common risk indicators, predict the trend of cluster-level load imbalance or data skew, and display the change impact path in the form of a parent-child tree to locate high-risk nodes and associated shards.

7. The control method for the change risk of the GoldenDB database according to claim 1, wherein In step S2, during the change parsing process, the rule library is used to integrate the data node parameter rules, computing node parameter rules, and middleware parameter rules, extract keywords through semantic parsing, and dynamically update the rule library to adapt to the GoldenDB version upgrade.

8. A control system for the change risk of the GoldenDB database, which applies the control method for the change risk of the GoldenDB database described in claim 1, characterized in that, The system includes: A change collection and parsing module, used to obtain and parse the change plan, and output the classified single transaction operation; An execution domain analysis module, used to determine the nodes and shard ranges affected by the change based on the pre-execution technology; A risk assessment module, including four sub-modules of permissions, parameters, data, and structure, used to perform multi-dimensional risk analysis on the computing nodes and data nodes respectively; A risk summary module, used to aggregate cross-node risk data, generate a control report, and trigger a control instruction.

9. The control system for the change risk of the GoldenDB database according to claim 8, characterized in that, The risk assessment module further includes: A real-time monitoring unit, used to collect node resource status, transaction conflict rate, and cluster health indicators; A dynamic blocking unit, used to automatically intercept high-risk changes according to the risk assessment results and notify the operation and maintenance personnel synchronously; A change backtracking unit, used to record all change operations, risk assessment processes, and control results, and support historical data query and auditing.

Images