A database updating method, device, electronic device and storage medium

Abstract

The present application relates to a database update method, device, electronic device and storage medium, wherein the database update method includes: based on a graph theory algorithm, a first database schema diagram of a source database and a second database schema diagram corresponding to a target database; generating multiple incremental execution scripts according to the first database schema diagram and the second database schema diagram; constructing a script execution diagram according to the incremental execution script, and determining the script execution order based on the topological structure of the script execution diagram; and executing the incremental execution script based on the script execution order. Through this application, the problem of how to ensure the consistency and stability of database updates through intelligent means, which is complicated and error-prone in version control during the update process of databases, especially database clusters, is solved, and the unification of operation logic and operation methods for different types of databases in a multi-client, multi-version, and multi-user environment is achieved, ensuring the consistency of database structure and data.

Description

Technical Field

[0001] The present application relates to the technical field of database processing, and in particular to a database updating method, device, electronic device and storage medium. Background Art

[0002] In today's era of rapid development of informatization, databases are the core infrastructure for data storage and management, and their importance is self-evident. With the widespread application of technologies such as cloud computing, big data, and the Internet of Things, the types and scale of databases are constantly expanding, from traditional relational databases to non-relational databases and time series databases, and the application scenarios of databases are becoming increasingly diverse. Therefore, in such a complex database environment, database updates and maintenance are crucial to ensure stable system operation and support business continuity.

[0003] Existing database update methods usually have the following problems:

[0004] ① Version control in the traditional database update process is complex and error-prone, and does not support multiple database types. Version control mainly relies on manual management, which is prone to version confusion, missed updates and other problems. Especially in a multi-version, multi-user environment, the complexity of version control increases significantly, and it is easy to introduce human errors. Moreover, traditional database update methods often only support specific database types and cannot adapt to diverse database update needs.

[0005] ② The traditional database update process lacks intelligence, making it difficult to ensure data consistency and stability. Most existing database update tools only provide basic script execution functions, cannot automatically identify differences between databases, and often find it difficult to fully consider various potential risks and dependencies, which can easily lead to data inconsistency or system instability.

[0006] ③ When faced with large-scale data updates, traditional serial update methods often have performance bottlenecks and cannot achieve the upgrade of large distributed database clusters. In addition, issues such as concurrency control and transaction management during the update process further increase the complexity of the update, which will lead to low update efficiency.

[0007] At present, for databases in related technologies, especially database clusters, version control is complex and error-prone during the update process. There is no effective solution to the problem of how to ensure the consistency and stability of database updates through intelligent means. Summary of the invention

[0008] The embodiments of the present application provide a database update method, apparatus, system, electronic device and storage medium to at least solve the problem in the related art that version control of databases, especially database clusters, is complex and error-prone during the update process, and how to ensure the consistency and stability of database updates through intelligent means.

[0009] In a first aspect, an embodiment of the present application provides a database updating method, the method comprising:

[0010] Acquire a first database object and a first object dependency relationship of a source database, and construct a first database schema diagram of the source database according to the first database object and the first object dependency relationship;

[0011] Acquire a second database object and a second object dependency relationship of a target database, and construct a second database schema diagram corresponding to the target database according to the second database object and the second object dependency relationship;

[0012] Determine the incremental information of the target database according to the first database schema diagram and the second database schema diagram; generate multiple incremental execution scripts based on the incremental information, and determine the script dependency relationship between the incremental execution scripts;

[0013] Constructing a script execution graph according to the incremental execution script and the script dependency, and determining a script execution order based on a topological structure of the script execution graph;

[0014] Based on the script execution order, the incremental execution script is executed.

[0015] In some embodiments, determining a script execution order based on a topological structure of the script execution graph, and executing the incremental execution script based on the script execution order includes:

[0016] Determining an initial script execution order based on the topological structure of the script execution graph;

[0017] Selecting a leader node and multiple other follower nodes in the execution script object based on the Raft algorithm;

[0018] Based on the script dependency and a preset script priority order, sending the incremental execution script to each of the follower nodes via the leader node;

[0019] The follower node sends the follower node information to the leader node, and the leader node optimizes the initial script execution order based on the follower node information to determine the script execution order;

[0020] Based on the script execution order, the incremental execution script is executed.

[0021] In some embodiments, based on the script execution order, after executing the incremental execution script, the method further includes:

[0022] If it is detected that the incremental execution script has been executed, the execution record of the completed script is recorded in the leader log, and the leader log is stored in the leader node;

[0023] If it is detected that the incremental execution script has not been executed, the follower node processes the unexecuted incremental execution script in response to the instruction of the leader node until the incremental execution script has been executed.

[0024] In some embodiments, executing the incremental execution script based on the script execution order further includes:

[0025] Script identification information of each incremental execution script is defined, and based on the script identification information, the plurality of incremental execution scripts are executed in parallel according to the script execution order.

[0026] In some embodiments, defining the script identification information of each incremental execution script further includes:

[0027] If the number of the target databases is at least two, each of the script identification information includes atomic statements of multiple databases, and the atomic statements are statements involving the same execution script object.

[0028] In some embodiments, executing the incremental execution script based on the script execution order includes:

[0029] Performing a preparation operation on the incremental execution script;

[0030] If all the preparation operations are executed normally, a confirmation operation is performed, wherein the confirmation operation is an operation of executing the incremental execution script based on the preparation operations and the script execution order, and recording the script execution status at the same time;

[0031] If the preparation operation fails to be executed, a cancellation operation is performed, wherein the cancellation operation is an operation of canceling the preparation operation executed on the incremental execution script;

[0032] Based on the cancel operation, it is queried whether the corresponding script identification information exists, and the script execution status is recorded at the same time.

[0033] In some embodiments, performing a preparation operation on the incremental execution script includes:

[0034] Detect the incremental execution scripts, add shared locks to non-conflicting incremental execution scripts, and add exclusive locks to dependent incremental execution scripts;

[0035] The script isolation level of the incremental execution script is determined according to a preset script isolation level type.

[0036] In some of the embodiments, the type of the target database includes a relational database, a non-relational database, and a time series database.

[0037] In a second aspect, an embodiment of the present application provides a database updating device, including:

[0038] A first graph building module, configured to obtain a first database object and a first object dependency relationship of a source database, and to build a first database schema graph of the source database according to the first database object and the first object dependency relationship;

[0039] A second graph building module, configured to obtain a second database object and a second object dependency relationship of at least one target database, and to build a first database schema graph corresponding to the target database according to the second database object and the second object dependency relationship;

[0040] A script generation module, configured to determine the incremental information of the target database according to the first database schema diagram and the second database schema diagram, and generate a plurality of incremental execution scripts based on the incremental information, wherein the incremental execution scripts include execution script objects and script dependency relationships;

[0041] An execution order generation module, used to construct a script execution graph according to the execution script object and the script dependency relationship, and determine the script execution order based on the topological structure of the script execution graph;

[0042] A script execution module is used to execute the incremental execution script based on the script execution order.

[0043] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the database updating method described in the first aspect above is implemented.

[0044] Compared with the related technologies, the database update method provided in the embodiment of the present application realizes the intelligent and automated database version management by automatically detecting, analyzing, executing and tracking database changes, solves the problems of data inconsistency, update failure, rollback difficulty and so on that may occur in the traditional database update process, and realizes the unification of the operation logic and operation methods of different types of databases. It aims to solve the problem of database structure and data consistency maintenance in a multi-client, multi-version and multi-user environment through an efficient, secure and traceable mechanism.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 is a hardware structure block diagram of a terminal according to a database updating method of an embodiment of the present application;

[0048] Figure 2 is a flowchart of a database updating method according to an embodiment of the present application;

[0049] Figure 3 is a preferred flow chart of a database updating method according to an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of a preferred structure of a database updating method according to an embodiment of the present application;

[0051] Figure 5 It is a structural block diagram of a database updating device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0053] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0054] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of a database updating method according to an embodiment of the present invention. Figure 1 As shown, the terminal 10 may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1Different configurations shown.

[0056] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the database update method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0057] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by the communication provider of the terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0058] This embodiment provides a database updating method. Figure 2 is a flowchart of a database updating method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0059] Step S201 is used to obtain a first database object and a first object dependency relationship of a source database, and construct a first database schema diagram of the source database according to the first database object and the first object dependency relationship.

[0060] The present invention can collect metadata information (Metadata) of the source database through the interface or tools provided by the database management system, including table structure, index, view, stored procedure, function, trigger, etc. Based on the graph theory algorithm, each database model is regarded as a graph, and the specific method is as follows:

[0061] Vertices: represent database objects, such as tables, views, stored procedures, functions, triggers, etc.

[0062] Edges: represent the dependency relationship between objects, such as foreign key-related tables, tables involved in triggers, etc.

[0063] Attributes: Record each object (such as field name, data type, whether it can be null, index, primary key, etc.) as a label or attribute attached to the node.

[0064] In this step, the first database object V1 is the vertex and its attribute information in the source database, and the first object dependency E1 is the edge in the source database. According to the first database object V1 and the first object dependency E1, the first database schema G1=(V1, E1) of the source database is constructed. In this step, the first database schema of the source database is constructed based on the collected metadata, so that the changes in the database can be detected, providing a complete data foundation, and representing the database structure in the form of a graph, which is easy to understand and analyze, and realizes the visualization of the database structure.

[0065] Step S202, obtaining the second database object and the second object dependency of the target database, and constructing a second database schema diagram corresponding to the target database according to the second database object and the second object dependency.

[0066] Among them, the target database is the new version that the source database needs to reach. In this embodiment, multiple databases can be updated in parallel, that is, the database cluster is updated, and multiple target databases can be determined from the database cluster. Similar to step S201, the second database object V2 is the vertex and its attribute information in the target database, and the second object dependency E2 is the edge in the target database. According to the second database object V2 and the second object dependency E2, the second database model graph G2=(V2, E2) of the target database is constructed, which reflects the structure of the target database and the dependency relationship between objects. This step visualizes the structure of the target database, making it easy to compare with the source database, identify incremental differences, generate update scripts to implement update operations, and support the construction of target databases of different versions or different configurations, ensuring the consistency of data during the update process of different types of databases.

[0067] Step S203, determining the incremental information of the target database according to the first database schema diagram and the second database schema diagram; generating multiple incremental execution scripts based on the incremental information, and determining the script dependency relationship between the incremental execution scripts.

[0068] Among them, this step compares the differences between G1 and G2, specifically, V1 and V2, to identify the newly added, deleted or modified objects and specific attributes; compares E1 and E2 to find the increase or decrease or change of dependencies, and obtains the difference between the target database and the source database regarding objects, specific attributes and dependencies between objects as incremental information. Based on the incremental information, an incremental execution script is generated according to the database-specific syntax. The incremental execution script consists of DDL (Data Definition Languages) / DML (Data Manipulation Language) statements.

[0069] The database-specific syntax is as follows:

[0070] When adding an object: Generate CREATE TABLE, CREATE VIEW, CREATE PROCEDURE and other statements;

[0071] When deleting an object: Generate DROP TABLE, DROP VIEW and other statements;

[0072] When modifying an object: Generate ALTER TABLE operations such as adding or deleting columns, modifying constraints, etc.

[0073] When re-establishing or deleting a dependency: Generate corresponding ALTER TABLE ADD FOREIGN KEY or DROPCONSTRAINT statements;

[0074] When attributes change: Generate INSERT, UPDATE, DELETE and other statements.

[0075] This step uses the subgraph isomorphism algorithm to analyze the differences in nodes, edges, and attributes, identify newly added, deleted, or modified objects and their attributes, as well as the increase, decrease, or change in dependencies, and automatically generates the script statements required for the update, avoiding the tediousness and error-proneness of manual script writing. At the same time, because the incremental execution script only contains necessary changes, it reduces data redundancy and potential risks during the update process.

[0076] Step S204: construct a script execution graph according to the incremental execution scripts and script dependencies, and determine the script execution order based on the topological structure of the script execution graph.

[0077] The script dependency is the sequential relationship between the incremental execution scripts (for example, a table must be created before a foreign key can be created). Each incremental execution script is regarded as a node, and the dependencies between scripts are regarded as edges to construct a script execution graph, which is a Directed Acyclic Graph (DAG).

[0078] Use the topological sorting algorithm to sort the script execution graph and generate a linear sequence as the script execution order, ensuring that each script is executed after the scripts it depends on. This step builds a DAG for the incremental execution scripts and topologically sorts the DAG to ensure the order and consistency of database updates. Topological sorting optimizes the execution order of the initial script, avoids unnecessary waiting and conflicts, and improves the efficiency of script execution.

[0079] Step S205: Execute the incremental execution script based on the script execution order.

[0080] Among them, according to the execution order determined by the above steps, multiple incremental execution scripts are executed. When all scripts are executed and successfully submitted, it means that the database update is completed, which realizes the intelligent update of the database and improves the automation and reliability of the update process.

[0081] Through the above steps, constructing the schema diagram of the source database and the target database can unify the form of data. No matter what kind of database it is, the data update status can be intuitively represented by constructing a schema diagram. The differences in the schema diagram can be compared to generate execution scripts, so that the update scripts can be automatically generated, avoiding the tediousness and error-proneness of manual script writing. Since the incremental script only contains necessary changes, it reduces data redundancy and potential risks in the update process and improves the stability of automated updates. For multiple execution scripts, a script execution diagram is constructed to ensure that the scripts are executed in the correct order and avoid execution failures caused by dependency errors. The use of topological sorting algorithms also optimizes the execution order and improves the efficiency of the update process. It solves the problem of how to use intelligent means to ensure the consistency and stability of database updates when version control is complex and error-prone during the update process of databases, especially database clusters. Through automation and intelligent methods, accurate, efficient and secure updates of database structures and data are achieved.

[0082] In some embodiments, based on the topological structure of the script execution graph, determining the script execution order, and executing the incremental execution script based on the script execution order, includes:

[0083] S241, determining an initial script execution order based on the topological structure of the script execution graph;

[0084] Among them, a DAG is constructed based on all scripts and their dependencies, in which nodes represent scripts and directed edges represent dependencies between scripts. Then a topological sorting algorithm (such as the Kahn algorithm or the non-recursive version of DFS post-order traversal) is used to sort the script execution graph to generate a linear script execution order that meets the dependency requirements of all scripts. This order is also the initial script execution order. In addition, if multiple scripts need to be executed concurrently, there are multiple paths, one of which is a critical path, which is used to determine the order of scheduled activities. For example, the critical path is to create a table, modify the table structure, create a view, and create a function. After these operations are completed, operations such as insert / update / delete can be performed. This step ensures that all scripts are executed correctly according to their dependencies by constructing DAG and topological sorting, avoiding execution errors or data inconsistencies. In addition, planning the execution order in advance can reduce waiting time and execution interruptions caused by dependency issues.

[0085] S242, selecting a leader node and multiple other follower nodes in the execution script object based on the Raft algorithm;

[0086] The nodes that execute the script incrementally form a Raft cluster. Each node is a potential leader or follower. The leader node is elected based on the Raft algorithm, which can usually be achieved through heartbeat messages and voting. In this step, a leader is elected through the Raft algorithm to coordinate the operations of the follower nodes to ensure the consistency of execution.

[0087] S243, based on the script dependency and the preset script priority order, the leader node sends the incremental execution script to each follower node; wherein the follower node sends the follower node information to the leader node, and the leader node optimizes the initial script execution order based on the follower node information and determines the script execution order;

[0088] Among them, first determine the dependency between scripts, and then pre-set the script priority order according to the importance, urgency and possible impact range of the script. In addition, the follower node must send its own resource occupancy and processing status to the leader node. The follower node information is the processing capacity, load, etc. of the follower node. The leader node further optimizes the initial script execution order based on the follower node information and the script dependency and priority order, thereby determining the script execution order. The leader node distributes the optimized script execution order and the corresponding incremental execution script to each follower node based on this script execution order. This step optimizes the execution order by considering the status of the follower node, so as to help balance the load. The system can adjust the execution plan according to the real-time status, thereby enhancing flexibility and adaptability and improving overall execution efficiency.

[0089] S244, executing the incremental execution script based on the script execution order.

[0090] Among them, while each follower node is performing a local update operation, it is also necessary to maintain communication with the leader node, so as to ensure the consistency of the operation. In addition, during the execution process, this embodiment supports the member changes of the node, and the failure or change of a single node will not affect the script execution process, thereby improving the stability of the database update. In this embodiment, all nodes (including leaders and followers) synchronously execute incremental scripts according to the optimized script execution order. During the execution process, maintaining communication between nodes can ensure that all nodes are executed to the same state and that the data on all nodes remain consistent. Compared with the prior art, it is impossible to dynamically generate the optimal update strategy. The leader node in the Raft algorithm of the above embodiment of this application can adjust the execution plan according to the real-time status of the follower node, thereby dynamically adjusting the update script and further optimizing the execution order, which can not only ensure the correct execution order and dependencies of the script, but also improve the reliability, consistency and flexibility of the system.

[0091] The implementation method based on script execution graph, Raft algorithm and script dependencies can not only ensure the correct execution order and dependencies of scripts, but also improve the reliability, consistency and flexibility of the system.

[0092] In some embodiments, after executing the incremental execution script based on the script execution order, the method further includes:

[0093] If it is detected that the incremental execution script has been executed, the execution record of the completed script is recorded in the leader log, which is stored in the leader node;

[0094] If it is detected that the incremental execution script has not been executed, the follower node responds to the instruction of the leader node and processes the incremental execution script that has not been executed until the incremental execution script has been executed.

[0095] Among them, when the leader node detects that an incremental execution script has been successfully executed, it will record the completion status of the script in its locally stored leader log, and add a record in the leader log file, which contains the identification information of the script (such as script ID, execution timestamp, etc.) and the execution result (success / failure). If the leader node detects that an incremental execution script has not been executed, it may be because some follower nodes have failed to execute or network problems have caused the message to be out of sync. It will send instructions to all follower nodes, asking them to retry to execute these unfinished scripts. After receiving the instructions from the leader node, the follower nodes will check their own execution status and re-execute those scripts marked as unfinished. Once the execution is completed, they will report the execution results to the leader node. This step can easily track the execution status and results of each script by recording logs, which is helpful for troubleshooting and performance analysis. For unexecuted scripts, the system can automatically retry, which improves the reliability and fault tolerance of execution. The logging function ensures the consistency of system data by ensuring that all nodes are executed to the same state.

[0096] In some embodiments, based on the script execution order, executing the script incrementally further includes:

[0097] Script identification information of each incremental execution script is defined, and based on the script identification information, multiple incremental execution scripts are executed in parallel according to the script execution order.

[0098] Among them, each incremental execution script generated by the algorithm can be defined as the script identification information V x.y.z.i , where x is the major version number, y is the minor version number / revision number, z is an increasing sequence, and i is the date version number plus the Greek letter version number. These scripts are executed in parallel on the leader node or follower node, and their execution status is tracked according to the script identification information. This step defines the script identification information, so that when abnormal execution is found later and a rollback operation is required, the version tracking during the script execution process can be performed, ensuring the stability of database updates.

[0099] In some embodiments, the script identification information of each incremental execution script is defined, further comprising:

[0100] If there are at least two target databases, each script identification information includes atomic statements of multiple databases, where the atomic statements are statements involving the same execution script object.

[0101] Among them, when defining the script identification information of each incremental execution script, if the system needs to process at least two target databases and needs to ensure that the operations of each execution script on multiple databases remain atomic, then atomic statements of multiple databases can be embedded in the script identification information. Atomicity refers to the indivisibility of a transaction. All operations of a transaction are either executed uninterruptedly or none of them are executed. Here, "atomic statements" refer to a set of SQL statements for the same execution script object (such as tables, views, etc.) that need to be executed synchronously on multiple databases and are indivisible. Each script identification information contains atomic statements of one or more databases. The SQL statements involving the same object are placed in an identification information file to ensure that they can be accurately identified and applied to the corresponding database during execution.

[0102] This step ensures the consistency of data between multiple databases by synchronously executing the atomic statement set of the same script object on these databases, avoiding data inconsistency problems caused by failure to update some databases. Setting script identification information to manage script update versions ensures the complete execution of the corresponding atomic statement set and improves the stability of the system. Even if a database has a problem during execution, it can be restored to a consistent state through a rollback mechanism. In addition, whether adding a new database or modifying the structure of an existing database, it can be achieved by updating the incremental execution script, making the database update system easier to expand and maintain.

[0103] In some embodiments, based on the script execution order, performing incremental execution of the script includes:

[0104] S251, perform preparation operations on the incremental execution script; wherein, the preparation operations are performed in a predetermined order, including detection of the incremental execution script, locking, and determination of the isolation level. If all preparation operations are successfully performed, proceed to the next step of confirmation operation.

[0105] S252, if all the preparation operations are executed normally, a confirmation operation is performed, and the confirmation operation is to execute the script incrementally based on the preparation operations and the script execution order, and record the script execution status at the same time; wherein, based on the results of the preparation operations and the script execution order, the incremental execution of the script is started, and the script execution status is recorded for subsequent monitoring and log analysis.

[0106] S253, if the preparation operation fails, a cancel operation is performed, and the cancel operation is an operation to cancel the preparation operation performed on the incremental execution script; based on the cancel operation, the corresponding script identification information is queried to see whether it exists, and the script execution status is recorded at the same time. Among them, if any failure is encountered during the execution of the preparation operation, a cancel operation is performed, and the cancel operation includes releasing the applied lock, rolling back the partially executed script, etc., to ensure the consistency of the system status. After the cancel operation, it is queried whether the corresponding script identification information exists, and the execution status of the script is updated to "failed" or "canceled" to avoid the suspension exception caused by the cancellation operation in the later stage being executed before the previous preparation operation, which is convenient for subsequent troubleshooting, system recovery and performance optimization.

[0107] In addition, in the present embodiment, when any step has an exception, a rollback operation needs to be performed. The rollback operation is to restore the database to execute the script to the version before the exception operation, and to query whether the corresponding script identification information exists to avoid an empty rollback phenomenon when the script identification information does not exist. The execution status of the script needs to be updated in time so that the update system does not reuse or release resources.

[0108] Figure 3 It is a preferred flow chart of the database updating method according to the embodiment of the present application, which is specifically described as follows:

[0109] Figure 3 The entire process of executing the database update transaction for this embodiment;

[0110] S301: Transaction initiator / committer 1 registers and initiates a global update transaction;

[0111] S302: Transaction initiator / committer 1 sends update transaction related information and executes global update transaction instruction to transaction initiator / committer 2 and transaction initiator / committer 3;

[0112] S303: All transaction initiators / committers register to initiate branch update transactions;

[0113] S304: The resource managers in all transaction initiators / committers perform a preparation operation (Try) of the branch update transaction on their respective connected databases;

[0114] S305: The transaction initiator / committer 1 confirms to the transaction coordinator that it has entered the second stage of TCC (Try / Confirm / Cancel), that is, confirming the operation or canceling the operation;

[0115] S306: The transaction coordinator notifies all transaction initiators / committers to perform confirmation / cancellation operations;

[0116] S307: The resource managers in all transaction initiators / committers perform confirmation / cancel operations on their respective connected databases;

[0117] S308: All transaction initiators / committers report the operation results to the transaction coordinator.

[0118] This embodiment uses the optimized TCRC (Try / Confirm / Rollback / Cancel) mode. Figure 3 Rollback operations are performed when abnormalities occur in the execution of each step, so that the update system can respond quickly and restore to a stable state when faced with abnormal situations. Detailed preparation operations are performed before incremental execution of scripts to protect the consistency and integrity of the data. The clear division and orderly execution of preparation operations, confirmation operations, and cancellation operations, and the recording of script execution status information provide an important basis for subsequent monitoring and management. Administrators can understand the system's operating status and make corresponding optimizations and adjustments by viewing these log information.

[0119] In some embodiments, performing a preparation operation on the incremental execution script includes:

[0120] S251-1, detect the incremental execution scripts, add shared locks to the incremental execution scripts without conflicts, and add exclusive locks to the incremental execution scripts with dependencies;

[0121] Among them, each incremental execution script is first tested to analyze whether there is a potential conflict in the dependencies between the scripts. For incremental execution scripts without conflicts, the system is ready to execute these scripts and consider adding shared locks when necessary. Shared locks allow other transactions to read data, but do not allow modifications, ensuring that the consistency of data is not destroyed during the execution process. For incremental execution scripts with dependencies, the system adds exclusive locks. Exclusive locks ensure that during execution, no other transaction can read or modify the data, thereby avoiding data competition and conflicts. This step detects and locks the database to avoid unnecessary operations, thereby reserving resources during the update process and protecting data consistency.

[0122] S251 - 2 , determining the script isolation level of the incremental execution script according to a preset script isolation level type.

[0123] Among them, the script isolation level types include low isolation level and high isolation level. The low isolation level includes: Read Uncommitted, Read Committed, Repeatable Read. The low isolation level may allow higher concurrency, but may introduce problems such as dirty reads, non-repeatable reads or phantom reads; high isolation levels (such as serializable) can provide stricter data consistency protection, but may reduce concurrency performance. Concurrency problems need to be completely avoided and must be executed in an optimized order. Therefore, scripts with high isolation levels need to be optimized based on the Raft algorithm for script execution order, and then executed in the optimized order. This embodiment performs detailed preparation operations (such as locking, determining isolation levels, etc.) before incrementally executing scripts. The reasonable setting of the script isolation level ensures that the concurrency performance is improved as much as possible while ensuring data consistency, reduces performance bottlenecks caused by data lock competition, and more effectively protects data consistency and integrity, and improves overall execution efficiency.

[0124] In some of the embodiments, the type of the target database includes but is not limited to various relational databases, non-relational databases, and time series databases. Figure 4 is a schematic diagram of a preferred structure of a database updating method according to an embodiment of the present application, such as Figure 4As shown, in the intelligent database update server, database model management is to manage different types of database models; data standard management is to define and enforce data standards to ensure data consistency and accuracy; ER diagram management is to manage script execution diagrams; the graph theory algorithm module is used to generate database model diagrams, thereby generating update scripts; DDL script generation is used to automatically generate data definition language (DDL) scripts, including modifying database structures; DML script generation is used to generate data manipulation language scripts, including inserting, updating or deleting data; branching and merging is when data needs to be changed to different branches or merged branch data in database version control or data synchronization; data version management is to manage script identification information, which is used to track the version history of data and support version rollback and comparison. In the intelligent database update engine, the database connection engine is used to connect various databases that need to be updated; the concurrency control module is used to manage the concurrent execution of database operations to ensure data consistency and integrity; the version comparison module is used to compare the differences between different versions of the database and support update and rollback operations; the exception handling module is used to handle exceptions in database operations, such as rolling back transactions, recording logs, etc.; script topology sorting: when executing multiple scripts, sorting is performed according to the dependencies between scripts to ensure that the scripts are executed in the correct order; the rollback module is used to undo the executed database operations and restore the database to the previous state; the script execution engine is responsible for executing the update execution script; the interactive interface is used to provide an interface for users to interact with the database system. Among them, mysql, oracle, DAMO, Hangao and Renmin University of China Jincang are relational databases; mongodb and redis are non-relational databases.

[0125] like Figure 4 , the intelligent database update server plays a core management role. It is responsible for interacting with the intelligent database update engine, triggering data update operations, and managing data consistency. The intelligent database update server is responsible for receiving update instructions from the server and actually executing database update operations. The connection engine of the database in this embodiment uses JDBC connection technology, which theoretically supports any database with a JDBC driver. JDBC (Java Database Connectivity) is a standard API (Application Programming Interface) of Java. It provides a unified way to access the database and has a plug-in mechanism, allowing additional plug-ins to be installed to expand its functions. Therefore, this embodiment supports online and offline upgrades; supports relational databases, non-relational databases, time series databases, etc.; supports small databases (such as virus libraries, desktop applications, applets, configuration information, etc.) of client (PC / Android / iOS) applications Upgrade; supports the upgrade of large distributed database clusters.

[0126] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0127] Collect database metadata information to form a database baseline and baseline version control. When the database changes, we need to re-obtain metadata information, compare it with the latest version of the architecture snapshot, and then extract incremental information.

[0128] The database schema is considered as a graph, and graph algorithms are introduced to analyze the changes of the incremental version of the database, extract the incremental execution script of the database, extract the schema information from the source database and the target database respectively, and construct two graphs G1=(V1, E1) and G2=(V2, E2). The vertex V represents a specific database object with corresponding attribute information, and E represents the object dependency. The subgraph isomorphism algorithm is used to analyze the differences in nodes, edges, and attributes, and generate DDL / DML scripts according to the database-specific syntax.

[0129] The result after algorithm processing can be converted into an execution script. In practice, we define each incremental execution script as V x.y.z.i , where x is the major version number, y is the minor version number / revision number, z is an increasing sequence, and i is the date version number plus the Greek letter version number. Each version number contains one or more atomic statements of the database. SQL statements involving the same object are placed in one version file to facilitate subsequent exception handling and rollback operations.

[0130] The execution scripts of the database obtained earlier are regarded as multiple nodes, and the dependencies are regarded as edges. A directed acyclic graph (DAG) is constructed, and the DAG is sorted using topological sorting (Topological Sorting). The nodes in the graph are arranged into a linear sequence. Through topological sorting, a feasible execution order can be determined. During the concurrent execution process, the critical path method is used to arrange and optimize the task order.

[0131] When scheduling update scripts, a dynamic programming method is used to optimize the execution order, that is, based on the preset update strategy and rules, the execution order of the database update scripts is intelligently sorted and scheduled. The dynamic programming method is as follows:

[0132] A leader is elected based on the Raft algorithm; the database execution script is submitted to the leader, and the script is distributed to follower nodes based on the script scheduling strategy of dependency analysis and priority sorting; the follower nodes report resource usage and processing status, and the script execution strategy is dynamically adjusted accordingly; all submitted scripts are recorded in the leader log; unsubmitted operations are handled by followers; during the execution process, node member changes are supported; when a version of the script sequence is submitted, it means that the version update is completed; when all operations are submitted, the database update is completed.

[0133] Use the optimized TCRC mode to provide exception handling and rollback mechanisms during database upgrades and support visual monitoring. The specific implementation steps are as follows:

[0134] 1. Perform a Try (prepare) operation, which is mainly to test the database and reserve resources (lock resources). Add shared locks for non-conflicting resources, and add exclusive locks for closely dependent scripts.

[0135] 2. To ensure the availability of the database during the update and improve the system's mean time between failures, we need to determine the transaction isolation level of the script in advance and divide the transaction isolation level of the script into two categories. The low isolation level includes: Read Uncommitted, Read Committed, and Repeatable Read, which are submitted to the database for processing; the high isolation level needs to completely avoid concurrency issues and must be executed sequentially, so the scheduling module will allocate strategies.

[0136] 3. If all the Try operations are executed normally, the execution unit will continue to execute Confirm to complete the complete business logic; if the Try phase fails, Cancel will be executed to release the reserved business resources.

[0137] 4. For exception handling, we introduce a script control table in the scheduling module. After Try is executed, a record will be inserted. When Cancel is required, it will query whether the corresponding script id exists to avoid the empty rollback phenomenon. At the same time, the execution status is recorded to ensure the idempotence of TCC (Try-Confirm-Cancel, a model for distributed transaction processing) execution (idempotence means that one or a series of operations can be executed multiple times, but the result status is the same as when it is executed only once, that is, the result of the operation is certain and consistent, and will not change due to multiple executions), and resources will not be reused or released. After Cancel is executed, an execution record is added to avoid hanging exceptions caused by the execution of the Cancel request in the later stage before the previous Try request.

[0138] This embodiment also provides a database updating device, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0139] Figure 5 is a structural block diagram of a database updating device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0140] The first graph building module 10 is used to obtain the first database object and the first object dependency relationship of the source database, and build a first database schema graph of the source database according to the first database object and the first object dependency relationship.

[0141] The second mapping module 20 is used to obtain a second database object and a second object dependency relationship of at least one target database, and construct a second database schema graph corresponding to the target database according to the second database object and the second object dependency relationship.

[0142] The script generation module 30 is used to determine the incremental information of the target database according to the first database schema diagram and the second database schema diagram, and generate multiple incremental execution scripts based on the incremental information. The incremental execution scripts include execution script objects and script dependencies.

[0143] The execution order generation module 40 is used to construct a script execution graph according to the execution script objects and script dependencies, and determine the script execution order based on the topological structure of the script execution graph.

[0144] The script execution module 50 is used to execute the incremental execution script based on the script execution order.

[0145] The execution order generation module 40 is further used to determine the initial script execution order based on the topological structure of the script execution graph;

[0146] Based on the Raft algorithm, a leader node and multiple follower nodes are selected in the execution script object;

[0147] Based on the script dependencies and the pre-set script priority order, the leader node sends the incremental execution script to each follower node;

[0148] The follower node sends the follower node information to the leader node, and the leader node optimizes the initial script execution order based on the follower node information and determines the script execution order.

[0149] The execution script execution module 50 is further used to record the completed script execution record in the leader log if it is detected that the incremental execution script has been executed, and the leader log is stored in the leader node;

[0150] If it is detected that the incremental execution script has not been executed, the follower node responds to the instruction of the leader node and processes the incremental execution script that has not been executed until the incremental execution script has been executed.

[0151] The script execution module 50 is further used to define script identification information of each incremental execution script, and execute multiple incremental execution scripts in parallel according to the script execution order based on the script identification information.

[0152] The execution script execution module 50 is also used for, if the number of target databases is at least two, each script identification information contains atomic statements of multiple databases, where the atomic statements are statements involving the same execution script object.

[0153] The above-mentioned execution script execution module 50 is also used to perform preparation operations on the incremental execution script;

[0154] If all preparation operations are executed normally, the confirmation operation is performed. The confirmation operation is to execute the script incrementally based on the preparation operations and the script execution sequence, and record the script execution status at the same time;

[0155] If the preparation operation fails, a cancel operation is performed, where the cancel operation is to cancel the preparation operation performed on the incremental execution script;

[0156] Based on the cancel operation, check whether the corresponding script identification information exists, and record the script execution status.

[0157] The above-mentioned execution script execution module 50 is also used to detect the incremental execution script, add a shared lock to the incremental execution script without conflict, and add an exclusive lock to the incremental execution script with dependency;

[0158] Determine the script isolation level for incremental script execution based on the preset script isolation level type.

[0159] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0160] In addition, in combination with the database updating method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the database updating methods in the above embodiment is implemented.

[0161] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A database updating method, characterized in that: include: Acquire a first database object and a first object dependency relationship of a source database, and construct a first database schema diagram of the source database according to the first database object and the first object dependency relationship; Acquire a second database object and a second object dependency relationship of a target database, and construct a second database schema diagram corresponding to the target database according to the second database object and the second object dependency relationship; Determining incremental information of the target database according to the first database schema diagram and the second database schema diagram; Based on the incremental information, a plurality of incremental execution scripts are generated, and script dependencies between the incremental execution scripts are determined; Constructing a script execution graph according to the incremental execution script and the script dependency, and determining a script execution order based on a topological structure of the script execution graph; Based on the script execution order, the incremental execution script is executed.

2. The database updating method according to claim 1, characterized in that: The step of determining a script execution order based on the topological structure of the script execution graph, and executing the incremental execution script based on the script execution order, includes: Determining an initial script execution order based on the topological structure of the script execution graph; Selecting a leader node and multiple other follower nodes in the incremental execution script based on the Raft algorithm; Based on the script dependency and a preset script priority order, sending the incremental execution script to each of the follower nodes via the leader node; The follower node sends the follower node information to the leader node, and the leader node optimizes the initial script execution order based on the follower node information to determine the script execution order; Based on the script execution order, the incremental execution script is executed.

3. The database updating method according to claim 2, characterized in that: After executing the incremental execution script based on the script execution order, the method further includes: If it is detected that the incremental execution script has been executed, the execution record of the completed script is recorded in the leader log, and the leader log is stored in the leader node; If it is detected that the incremental execution script has not been executed, the follower node processes the incremental execution script that has not been executed in response to the instruction of the leader node until the incremental execution script has been executed.

4. The database updating method according to claim 1, characterized in that: The step of executing the incremental execution script based on the script execution order further includes: Script identification information of each incremental execution script is defined, and based on the script identification information, the plurality of incremental execution scripts are executed in parallel according to the script execution order.

5. The database updating method according to claim 4, characterized in that: The script identification information defining each incremental execution script further includes: If the number of the target databases is at least two, each of the script identification information includes atomic statements of multiple databases, and the atomic statements are statements related to the same incremental execution script.

6. The database updating method according to claim 4, characterized in that: The step of executing the incremental execution script based on the script execution order includes: Performing a preparation operation on the incremental execution script; If all the preparation operations are executed normally, a confirmation operation is performed, wherein the confirmation operation is an operation of executing the incremental execution script based on the preparation operations and the script execution order, and recording the script execution status at the same time; If the preparation operation fails to be executed, a cancellation operation is performed, wherein the cancellation operation is an operation of canceling the preparation operation executed on the incremental execution script; Based on the cancel operation, it is queried whether the corresponding script identification information exists, and the script execution status is recorded at the same time.

7. The database updating method according to claim 6, characterized in that: The step of preparing the incremental execution script includes: Detect the incremental execution scripts, add shared locks to non-conflicting incremental execution scripts, and add exclusive locks to dependent incremental execution scripts; According to a preset script isolation level type, the script isolation level of the incremental execution script is determined.

8. The database updating method according to any one of claims 1 to 7, characterized in that: The types of the target database include relational database, non-relational database, and time series database.

9. A database updating device, characterized in that: include: A first graph building module, configured to obtain a first database object and a first object dependency relationship of a source database, and to build a first database schema graph of the source database according to the first database object and the first object dependency relationship; A second graph building module, used for acquiring a second database object and a second object dependency relationship of a target database, and building a second database schema graph corresponding to the target database according to the second database object and the second object dependency relationship; A script generation module, used to determine the incremental information of the target database according to the first database schema diagram and the second database schema diagram, and generate a plurality of incremental execution scripts based on the incremental information, and determine the script dependency relationship between the incremental execution scripts; An execution order generation module, used to construct a script execution graph according to the incremental execution script and the script dependency relationship, and determine the script execution order based on the topological structure of the script execution graph; A script execution module is used to execute the incremental execution script based on the script execution order.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the database updating method according to any one of claims 1 to 8 when running.

Images