Database processing method, electronic device, storage medium, and program product
By automatically detecting and retrying DDL statements in a distributed database, the inefficiency of DDL operations in existing technologies is solved, achieving more efficient database processing.
Patent Information
- Application Number
- PCT/CN2024/142531
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-05-08
- Filing Date
- 2024-12-25
- Publication Date
- 2025-11-13
AI Technical Summary
Existing technologies require manual inspection and retrying when errors occur during Data Definition Language (DDL) operations in distributed database clusters, resulting in low database processing efficiency.
By receiving DDL requests, the system automatically detects statements that fail to execute, records the number of retries, and, if the table structure information allows, automatically retryes the DDL statements on other data nodes until the preset number of retries is reached or the transaction succeeds.
It simplifies the DDL operation process, improves the fault tolerance and efficiency of database processing, and reduces manual intervention.
Smart Images

Figure CN2024142531_13112025_PF_FP_ABST
Abstract
Description
Database processing methods, electronic devices, storage media and program products
[0001] This application claims priority to Chinese Patent Application No. 202410566280.6, filed on May 8, 2024, entitled "Database Processing Method, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of database technology, and in particular to a database processing method, electronic device, storage medium, and program product. Background Technology
[0003] In the process of handling transfer services, database processing operations are required, such as switching the database that carries the transfer services from a centralized database to a target distributed database.
[0004] In existing technologies, after a distributed database cluster identifies a Data Definition Language (DDL) operation request, it uses a third-party component to distribute the Structured Query Language (SQL) statements in the DDL operation request to the actual data nodes to execute the corresponding database processing operations.
[0005] However, the above method, which introduces third-party components, requires manual execution of a series of checks when DDL operations fail, which is cumbersome and reduces database processing efficiency. Summary of the Invention
[0006] This application provides a database processing method, electronic device, storage medium, and program product to improve database processing efficiency.
[0007] In a first aspect, embodiments of this application provide a database processing method, including:
[0008] The system receives a Data Definition Language (DDL) request sent by an application device, and executes each DDL statement in the DDL request based on at least one first data node to obtain a first execution result for each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents database table operations used to process each database.
[0009] If the first execution result indicates that the DDL statement has failed, then the execution status of the DDL statement is updated to a failed state; and the number of retries for the DDL statement is determined; wherein, the number of retries represents the total number of times the DDL statement is re-executed;
[0010] If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the table information of the target database is determined, and the DDL statement is executed again based on the table information of the target database to obtain the second execution result of the DDL statement; wherein, the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database.
[0011] The second execution result of the DDL statement is sent to the application device.
[0012] In one possible implementation, based on the table information of the target database, the DDL statement is executed again to obtain a second execution result of the DDL statement, including:
[0013] Based on the metadata node, and according to the table information of the target database, the second data node corresponding to the DDL statement is determined; wherein, the second data node is a first data node other than the first data node that successfully executed the DDL statement;
[0014] The DDL statement is sent to the second data node corresponding to the DDL statement; wherein the second data node is used to re-execute the DDL statement to obtain the second execution result.
[0015] In one possible implementation, based on the metadata node, the second data node corresponding to the DDL statement is determined according to the table information of the target database, including:
[0016] The table information is processed to obtain an identification result; wherein, the identification result represents the current version status of the table information;
[0017] If it is determined that the identification result indicates that the table information is not the latest version, then the second data node is determined.
[0018] In one possible implementation, the table information includes field information, which characterizes the field types and lengths of the table structure; the table information is subjected to identification processing to obtain an identification result, including:
[0019] The field information in the table is identified to obtain the identification result.
[0020] In one possible implementation, after processing the table information to obtain the identification result, the method further includes:
[0021] If the identification result indicates that the table information is the latest version, then the first execution result of the DDL statement indicates that the DDL statement was executed successfully.
[0022] The first execution result of the DDL statement is sent to the application device.
[0023] In one possible implementation, based on at least one first data node, each DDL statement in the DDL request is executed to obtain a first execution result for each DDL statement, including:
[0024] Based on the metadata node, a DDL task corresponding to the DDL request is generated; wherein, the DDL task includes at least one DDL statement from the DDL request; the DDL task has task information; the task information represents the execution status of each DDL statement in the DDL task;
[0025] Write the task information of each DDL task into the subtask table;
[0026] Based on each of the first data nodes, the DDL statements in the DDL task are executed to obtain the first execution result of the DDL statements.
[0027] In one possible implementation, based on the metadata node, a DDL task corresponding to the DDL request is generated, including:
[0028] Based on the metadata node, each DDL statement in the DDL request is encapsulated to obtain the DDL job corresponding to the DDL request; wherein, the DDL job has job information; the job information is the database table information to be processed corresponding to the DDL job;
[0029] If it is determined that the DDL job is allowed to be executed, the job information is stored in a metadata table; wherein, the metadata table includes a job table and an information table; the job table is used to record the execution status of currently executing or unexecuted DDL statements; the information table is used to record the changes in the target database before and after the execution of the DDL statement;
[0030] The number of nodes is determined, and each DDL task is generated based on the number of nodes; wherein the number of nodes is the total number of first data nodes that can be used to perform database operations; and each DDL task corresponds one-to-one with the first data node.
[0031] In one possible implementation, determining that the DDL job is allowed to be executed includes:
[0032] If it is determined that the target DDL job does not exist in the job table, then the DDL job is determined to be allowed to be executed; wherein, the target DDL job is a DDL job whose job information is consistent with the job information of the DDL job.
[0033] In one possible implementation, the information table includes table change information and each DDL statement in the DDL job; the table change information characterizes the changes in the table structure of the target database before and after the execution of the DDL statements.
[0034] In one possible implementation, the task information includes node information, database information, and log information; the node information is the data node information corresponding to the execution of the DDL task; the database information is the connection identifier information of the target database corresponding to the DDL task; and the log information is the error information generated during the execution of the DDL task.
[0035] In one possible implementation, sending the second execution result of the DDL statement to the application device includes:
[0036] Based on the second execution result of the DDL statement, the execution status of the DDL statement is updated.
[0037] If it is determined that the execution status of each DDL statement has been updated and processed, the second execution result of the DDL statement is sent to the application device.
[0038] In one possible implementation, updating the execution status of the DDL statement based on the second execution result includes:
[0039] If it is determined that the second execution result indicates that the DDL statement was executed successfully, then the execution status of the DDL statement is updated to a successful status;
[0040] If the second execution result indicates that the DDL statement has failed to execute, then the number of retries for the DDL statement is determined; and the execution status of the DDL statement is updated according to the number of retries for the DDL statement; wherein the number of retries represents the total number of times the DDL statement has been re-executed.
[0041] In one possible implementation, updating the execution status of the DDL statement based on the number of retries includes:
[0042] If it is determined that the number of retries for the DDL statement is greater than the preset number, the execution status of the DDL statement is updated to a failed state; and a prompt message is generated; wherein, the prompt message indicates that the current execution of the DDL statement has failed, and the prompt message is used to prompt manual handling;
[0043] If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the execution status of the DDL statement is updated to the in-process status; and the step of determining the table information of the target database is executed.
[0044] In one possible implementation, before determining the table information of the target database, the method further includes:
[0045] If it is determined that the DDL statement has connection information, then the connection information of the DDL statement is deleted; wherein, the connection information represents the connection relationship between the DDL statement and the first data node that last executed the DDL statement.
[0046] In one possible implementation, the method further includes:
[0047] If the first execution result indicates that the DDL statement was executed successfully, the execution status of the DDL statement is updated to a successful status; and the first execution result of the DDL statement is sent to the application device.
[0048] In one possible implementation, the method further includes:
[0049] If it is determined that the number of retries for the DDL statement is greater than the preset number, then the first execution result of the DDL statement is sent to the application device.
[0050] Secondly, embodiments of this application provide a database processing apparatus, including:
[0051] The first execution unit is configured to receive a Data Definition Language (DDL) request sent by an application device, and execute each DDL statement in the DDL request based on at least one first data node to obtain a first execution result for each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents database table operations used to process each database.
[0052] The determining unit is configured to update the execution status of the DDL statement to a failed state if the first execution result indicates that the DDL statement has failed to execute; and to determine the number of retries for the DDL statement; wherein the number of retries indicates the total number of times the DDL statement has been re-executed.
[0053] The second execution unit is configured to, if it is determined that the number of retries for the DDL statement is less than or equal to a preset number, determine the table information of the target database, and execute the DDL statement again based on the table information of the target database to obtain the second execution result of the DDL statement; wherein, the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database.
[0054] The sending unit is used to send the second execution result of the DDL statement to the application device.
[0055] In one possible implementation, the second execution unit is specifically used for:
[0056] Based on the metadata node, and according to the table information of the target database, the second data node corresponding to the DDL statement is determined; wherein, the second data node is a first data node other than the first data node that successfully executed the DDL statement;
[0057] The DDL statement is sent to the second data node corresponding to the DDL statement; wherein the second data node is used to re-execute the DDL statement to obtain the second execution result.
[0058] In one possible implementation, the second execution unit is specifically used for:
[0059] The table information is processed to obtain an identification result; wherein, the identification result represents the current version status of the table information;
[0060] If it is determined that the identification result indicates that the table information is not the latest version, then the second data node is determined.
[0061] In one possible implementation, the table information includes field information, which characterizes the field types and lengths of the table structure; the second execution unit is specifically used for:
[0062] The field information in the table is identified to obtain the identification result.
[0063] In one possible implementation, the second execution unit is further specifically used for:
[0064] If the identification result indicates that the table information is the latest version, then the first execution result of the DDL statement indicates that the DDL statement was executed successfully.
[0065] The first execution result of the DDL statement is sent to the application device.
[0066] In one possible implementation, the first execution unit is specifically used for:
[0067] Based on the metadata node, a DDL task corresponding to the DDL request is generated; wherein, the DDL task includes at least one DDL statement from the DDL request; the DDL task has task information; the task information represents the execution status of each DDL statement in the DDL task;
[0068] Write the task information of each DDL task into the subtask table;
[0069] Based on each of the first data nodes, the DDL statements in the DDL task are executed to obtain the first execution result of the DDL statements.
[0070] In one possible implementation, the first execution unit is specifically used for:
[0071] Based on the metadata node, each DDL statement in the DDL request is encapsulated to obtain the DDL job corresponding to the DDL request; wherein, the DDL job has job information; the job information is the database table information to be processed corresponding to the DDL job;
[0072] If it is determined that the DDL job is allowed to be executed, the job information is stored in a metadata table; wherein, the metadata table includes a job table and an information table; the job table is used to record the execution status of currently executing or unexecuted DDL statements; the information table is used to record the changes in the target database before and after the execution of the DDL statement;
[0073] The number of nodes is determined, and each DDL task is generated based on the number of nodes; wherein the number of nodes is the total number of first data nodes that can be used to perform database operations; and each DDL task corresponds one-to-one with the first data node.
[0074] In one possible implementation, the first execution unit is specifically used for:
[0075] If it is determined that the target DDL job does not exist in the job table, then the DDL job is determined to be allowed to be executed; wherein, the target DDL job is a DDL job whose job information is consistent with the job information of the DDL job.
[0076] In one possible implementation, the information table includes table change information and each DDL statement in the DDL job; the table change information characterizes the changes in the table structure of the target database before and after the execution of the DDL statements.
[0077] In one possible implementation, the task information includes node information, database information, and log information; the node information is the data node information corresponding to the execution of the DDL task; the database information is the connection identifier information of the target database corresponding to the DDL task; and the log information is the error information generated during the execution of the DDL task.
[0078] In one possible implementation, the transmitting unit is specifically used for:
[0079] Based on the second execution result of the DDL statement, the execution status of the DDL statement is updated.
[0080] If it is determined that the execution status of each DDL statement has been updated and processed, the second execution result of the DDL statement is sent to the application device.
[0081] In one possible implementation, the transmitting unit is specifically used for:
[0082] If it is determined that the second execution result indicates that the DDL statement was executed successfully, then the execution status of the DDL statement is updated to a successful status;
[0083] If the second execution result indicates that the DDL statement has failed to execute, then the number of retries for the DDL statement is determined; and the execution status of the DDL statement is updated according to the number of retries for the DDL statement; wherein the number of retries represents the total number of times the DDL statement has been re-executed.
[0084] In one possible implementation, the transmitting unit is specifically used for:
[0085] If it is determined that the number of retries for the DDL statement is greater than the preset number, the execution status of the DDL statement is updated to a failed state; and a prompt message is generated; wherein, the prompt message indicates that the current execution of the DDL statement has failed, and the prompt message is used to prompt manual handling;
[0086] If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the execution status of the DDL statement is updated to the in-process status; and the step of determining the table information of the target database is executed.
[0087] In one possible implementation, before the second execution unit specifically determines the table information of the target database, the apparatus is further configured to:
[0088] If it is determined that the DDL statement has connection information, then the connection information of the DDL statement is deleted; wherein, the connection information represents the connection relationship between the DDL statement and the first data node that last executed the DDL statement.
[0089] In one possible implementation, the device is further used for:
[0090] If the first execution result indicates that the DDL statement was executed successfully, the execution status of the DDL statement is updated to a successful status; and the first execution result of the DDL statement is sent to the application device.
[0091] In one possible implementation, the device further includes:
[0092] If it is determined that the number of retries for the DDL statement is greater than the preset number, then the first execution result of the DDL statement is sent to the application device.
[0093] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0094] The memory stores computer-executed instructions;
[0095] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0096] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0097] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0098] The database processing method, electronic device, storage medium, and program product provided in this application embodiment directly execute DDL statements at a single point in multiple databases. When a DDL statement execution fails, the DDL statement is automatically re-executed based on the total number of times the DDL statement is re-executed and the table structure information of the database to be processed, in order to complete the database processing operation. Thus, the process can be simplified, fault tolerance can be improved, and database processing efficiency can be improved. Attached Figure Description
[0099] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0100] Figure 1 is a schematic diagram of an application scenario provided in this application;
[0101] Figure 2 is a flowchart illustrating a database processing method provided in this application;
[0102] Figure 3 is a schematic diagram of the framework of a DDL module provided in this application;
[0103] Figure 4 is a flowchart illustrating another database processing method provided in this application;
[0104] Figure 5 is a schematic diagram of the execution flow of a DDL module provided in this application;
[0105] Figure 6 is a flowchart illustrating an exception handling mechanism provided in this application;
[0106] Figure 7 is a schematic diagram of the structure of a database processing device provided in this application;
[0107] Figure 8 is a schematic diagram of the structure of an electronic device provided in this application.
[0108] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0109] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.
[0110] First, let me explain the terms used in this application:
[0111] Data Definition Language (DDL or ddl) is a computer language used to define the data structure and architecture of a database. DDL is used to create, modify, and delete database objects such as tables, views, and indexes. Through DDL, users can define the attributes and relationships of database objects, such as data types, constraints, and relationships, thereby defining the structure and organization of the database.
[0112] Structured Query Language (SQL) is a standardized language used to manage relational databases. It is used to define, manipulate, and manage data in a database, including operations such as inserting, updating, deleting, and querying data.
[0113] In the process of handling transfer services, database processing operations are required, such as switching the database that carries the transfer services from a centralized database to a target distributed database.
[0114] In one example, the solution of executing through direct connection to cluster data nodes is suitable for distributed databases based on middleware architecture. In this architecture, the data sharding logic is recorded through middleware configuration. Through manual processing or external tools, each node of the distributed database cluster is accessed to execute DDL statements and confirm successful execution. At the same time, it is necessary to modify the data sharding part of the middleware configuration regarding the new version table structure.
[0115] However, the above method requires direct connection to all data nodes to execute DDL database operations and synchronous modification of the database sharding configuration related to the new version table structure. It does not support single points of failure and is not transparent to operations and maintenance personnel; it is also incompatible with all types of DDL statements.
[0116] In another example, by introducing a third-party component to assist execution, this approach is suitable for distributed databases with an architecture that includes a scheduling center and a metadata maintenance component. After recognizing a DDL operation, the distributed database cluster saves it as a task in the component that maintains metadata information. The cluster's management and scheduling center then pulls the DDL task from this component and distributes the relevant SQL statements to the actual data nodes for execution. Because of the scheduling center, it avoids directly connecting to each node to execute statements separately.
[0117] However, the above approach requires the introduction of third-party open-source components to maintain metadata and task information, which increases the complexity of the architecture and brings additional costs to maintaining the components. DDL execution on data shard nodes may encounter errors. The handling steps for DDL failures are complex and difficult to execute. A series of manual operations are required to check the current table structure and the execution status of the current task before formulating a retry plan, which leads to a decrease in database processing efficiency.
[0118] The database processing method provided in this application automatically re-executes the DDL statement when it fails, based on the total number of times the DDL statement has been re-executed and the table structure information of the database to be processed. This solves the technical problem of reduced database processing efficiency caused by the need to manually perform a series of operations to check the current table structure and the execution status of the current task before formulating a retry plan.
[0119] Figure 1 is a schematic diagram of an application scenario provided by this application. As shown in Figure 1, the specific application scenario of this application includes electronic device 101 and application device 102. In the process of processing the transfer service, the user sends a DDL request to electronic device 101 through the application device to request database processing operations. For example, if the database carrying the transfer service needs to be switched from a centralized database to a target distributed database, electronic device 101 responds to the DDL request and performs database operations on the centralized database.
[0120] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0121] Figure 2 is a flowchart illustrating a database processing method provided in this application. As shown in Figure 2, the method includes:
[0122] 201. Receive a Data Definition Language (DDL) request sent by an application device, and execute each DDL statement in the DDL request based on at least one first data node to obtain the first execution result of each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents the database table operations used to process each database.
[0123] In this embodiment, the executing entity is an electronic device. For ease of description, the term "device" will be used to refer to an electronic device in the following description. An electronic device can be any physical or virtual device capable of database processing, or it can be a terminal device that interacts with the user.
[0124] For example, when a database carrying a switching service needs to be switched from a centralized database to a target distributed database, the user sends a DDL request to the device via an application device to request processing of at least one database carrying the switching service. These databases store financial business data. The device receives the DDL request from the application device, parses the DDL request, and obtains each DDL statement in the request. Each DDL statement represents a database table operation used to process each database. The device sends each DDL statement to one or more first data nodes, executes each DDL statement based on each first data node, and obtains the execution result of each first data node executing the DDL statement. Based on the execution results of each first data node executing the DDL statement, the first execution result of each DDL statement can be obtained, determining whether each DDL statement was executed successfully. If each first data node executes the DDL statement successfully, the first execution result indicates that the DDL statement was executed successfully.
[0125] For example, Figure 3 is a schematic diagram of the framework of a DDL module provided in this application. As shown in Figure 3, the application connection proxy sends a DDL request. After parsing the SQL statement, the proxy recognizes that it is a DDL request and forwards the SQL statement in the DDL request to the metadata node. Each metadata node has master, slave, standby and other modes. The service server layer of each metadata node pushes the information after parsing the SQL statement down to the storage engine layer. The storage engine layer schedules the DDL job through the DDL scheduling module, that is, sends the DDL statement to the DDL execution module. The DDL execution module sends the DDL statement to the corresponding data node, such as data node 1, data node 2, ..., data node N. Each data node has master, slave, standby and other modes to execute the DDL statement and obtain the first execution result of each DDL statement. Each data node sends the first execution result of each DDL statement to the metadata node for processing.
[0126] 202. If the first execution result indicates that the DDL statement has failed, then update the execution status of the DDL statement to the failure status; and determine the number of retries for the DDL statement; where the number of retries represents the total number of times the DDL statement is re-executed.
[0127] For example, the most common exception scenario in distributed DDL is that when multiple data nodes execute DDL tasks, some data nodes execute successfully while others fail, which can lead to inconsistencies in the table structure among the data nodes. For each DDL statement, when any data node encounters an error while executing the DDL statement, i.e., when the device determines that the DDL statement has failed, an exception handling mechanism is triggered. Specifically, referring to Figure 3, for each failed DDL statement, the execution module in the device updates the execution status of the DDL statement to a failed state and determines the number of retries for the DDL statement, i.e., the total number of times the DDL statement will be re-executed, for further processing.
[0128] 203. If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the table information of the target database is determined, and the DDL statement is executed again based on the table information of the target database to obtain the second execution result of the DDL statement; where the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database.
[0129] For example, the device compares the number of retries for a failed DDL statement with a preset number. If the number of retries for the DDL statement is less than or equal to the preset number, or if the maximum number of retries has been exceeded, the device determines the table structure information of the target database corresponding to the DDL statement. Based on the table information of the target database, a table structure consistency check is performed. If the table structure consistency is successful, the DDL statement is sent to the corresponding data node and re-executed to obtain a second execution result of the DDL statement, thus determining whether the DDL statement was executed successfully. This allows for automatic retries, simplifying the process and improving fault tolerance.
[0130] 204. Send the second execution result of the DDL statement to the application device.
[0131] For example, referring to Figure 3, after the data node deployed on the device obtains the second execution result of each DDL statement, it returns the second execution result of each DDL statement, i.e. the final execution result, to the Server layer of the Metadata node. The Server layer replies to the Proxy, and then returns the final execution result of each DDL statement in the DDL request to the application on the application device.
[0132] The database processing method provided in this application embodiment directly executes DDL statements at a single point in multiple databases. When a DDL statement execution fails, it automatically re-executes the DDL statement based on the total number of times the DDL statement has been re-executed and the table structure information of the database to be processed, thereby completing the database processing operation; thus, it can improve the efficiency of database processing.
[0133] Figure 4 is a flowchart illustrating another database processing method provided in this application. As shown in Figure 4, this embodiment describes the database processing method in detail based on the embodiment in Figure 2. The method includes:
[0134] 301. Receive a Data Definition Language (DDL) request sent by the application device, wherein the DDL request is used to request processing of at least one database; the database is used to store financial business data.
[0135] For example, the content of this step can be found in step 201, and will not be repeated here.
[0136] 302. Based on the metadata node, generate a DDL task corresponding to the DDL request; wherein, the DDL task includes at least one DDL statement in the DDL request; the DDL task has task information; the task information represents the execution status of each DDL statement in the DDL task.
[0137] The task information includes node information, database information, and log information; the node information is the data node information corresponding to the execution of the DDL task; the database information is the connection identifier information of the target database corresponding to the DDL task; and the log information is the error information generated during the execution of the DDL task.
[0138] For example, Figure 5 is a schematic diagram of a DDL module execution flow provided in this application. As shown in Figure 5, based on the DDL module deployed on the device, the DDL scheduling module constructs a task list corresponding to each data node. Each task in the task list includes at least one DDL statement from the received DDL request. Each DDL task has task information. The DDL execution module adds the task list to the global task queue for processing. Each task information represents the execution status of each DDL statement in the DDL task. Each task information includes node information, database information, and log information. Each node information is the data node information corresponding to the executed DDL task, such as a data node identifier. Each database information is the connection identifier information of the target database corresponding to the DDL task, such as a database connection identifier. Each log information is the error information generated during the execution of the DDL task, such as an error code.
[0139] In one possible implementation, 302 includes:
[0140] The first step is to encapsulate each DDL statement in the DDL request based on the metadata node to obtain the DDL job corresponding to the DDL request. The DDL job has job information, which is the database table information to be processed corresponding to the DDL job.
[0141] The second step is to store the job information in the metadata table if it is determined that the DDL job can be executed. The metadata table includes a job table and an information table. The job table is used to record the execution status of DDL statements that are currently being executed or have not been executed. The information table is used to record the changes in the target database before and after the execution of the DDL statement.
[0142] The third step is to determine the number of nodes and generate each DDL task based on the number of nodes. The number of nodes is the total number of first data nodes that can be used to perform database operations. Each DDL task corresponds to a first data node.
[0143] The information table includes table change information and each DDL statement in the DDL job; the table change information represents the changes in the table structure of the target database before and after the execution of the DDL statement.
[0144] For example, referring to Figures 3 and 5, the Server layer of the Metadata node in the device pushes down the SQL parsed information to the storage engine layer. The storage engine layer encapsulates and processes each DDL statement in the DDL request through the DDL scheduling module to obtain the DDL job corresponding to the DDL request. If the current DDL job is allowed to execute, the job information is written, that is, the job information of the DDL job is stored in the metadata table. The job information is the database table information to be processed corresponding to the DDL job. The metadata table includes a job table and an information table. The job table is used to record the execution status of the DDL statements that are currently being executed or not executed. The information table is used to record the changes in the target database before and after the execution of the DDL statement, including changes in the table structure before and after execution, the DDL execution statement, etc. After the device's DDL scheduling module writes the DDL job information into the DDL job table and the DDL information table, it determines the total number of first data nodes that can be used to execute database operations, that is, the number of nodes. Based on the obtained number of nodes, a task list corresponding one-to-one with the data nodes is constructed, that is, one task corresponds to one data node.
[0145] In one possible implementation, the second step in 302, determining that a DDL job is allowed to be executed, includes: if it is determined that a target DDL job does not exist in the job table, then determining that the DDL job is allowed to be executed; wherein, the target DDL job is a DDL job whose job information is consistent with the job information of a DDL job.
[0146] Specifically, after the device schedules a DDL job, it needs to check the current DDL job, that is, to determine whether there is a DDL job in the job table whose job information is consistent with the job information of the DDL job, i.e., the target DDL job. If it is determined that there is no target DDL job in the job table, such as no DDL job in the job table that processes the operation statement of the same data table, then it is determined that the DDL job can be executed.
[0147] 303. Write the task information of each DDL task into the subtask table.
[0148] For example, referring to Figure 5, based on the DDL module deployed on the device, the DDL scheduling module writes the task information of each DDL task into a subtask table. The DDL scheduling module synchronously checks the status of all subtasks, that is, the execution status of each DDL statement executed by each data node in the DDL task. The scheduling module and execution module are implemented in the database kernel, thus directly inheriting the database kernel's parsing results for SQL statements. The parsing results are in the storage engine-related DDL classes. The scheduling module can obtain all DDL-related information and implement DDL-related interfaces under the distributed architecture; this mechanism ensures that this solution is highly compatible with existing DDL syntax.
[0149] 304. Based on each first data node, execute the DDL statements in the DDL task to obtain the first execution result of the DDL statements.
[0150] For example, referring to Figure 5, the thread group of the DDL execution module deployed on the device concurrently retrieves DDL tasks from the task queue, sends the DDL statements in each DDL task to each data node for execution, obtains the execution status of each DDL statement on each data node, and thus obtains the first execution result of each DDL statement. Based on the execution status of each DDL statement on each data node, the execution status of each DDL statement in the subtask table is updated. Furthermore, the DDL scheduling module and execution module are implemented in the database kernel, collaboratively implementing single-point DDL in a distributed database without introducing third-party components, avoiding the complex operations of directly connecting to each node for execution.
[0151] 305. If the first execution result indicates that the DDL statement has failed, then update the execution status of the DDL statement to the failure status; and determine the number of retries for the DDL statement; where the number of retries represents the total number of times the DDL statement is re-executed.
[0152] For example, the content of this step can be found in step 202, and will not be repeated here.
[0153] In one possible implementation, after step 303, the method further includes: if it is determined that the first execution result indicates that the DDL statement was executed successfully, then updating the execution status of the DDL statement to a successful status; and sending the first execution result of the DDL statement to the application device.
[0154] For example, for each DDL statement, if the device determines that the DDL statement has been executed successfully, it updates the execution status of the DDL statement to a successful status and, based on the metadata node, directly sends the first execution result of the successful execution of the DDL statement to the application device.
[0155] 306. If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then determine the table information of the target database.
[0156] For example, the content of this step can be found in step 203, and will not be repeated here.
[0157] In one possible implementation, before determining the table information of the target database, the method further includes: if it is determined that the DDL statement has connection information, then deleting the connection information of the DDL statement; wherein the connection information represents the connection relationship between the DDL statement and the first data node that last executed the DDL statement.
[0158] For example, for a failed DDL statement, if the device determines that the number of retries for the DDL statement is less than or equal to a preset number, it re-enters the distribution processing flow. First, it determines whether the DDL statement has connection information, that is, whether a connection exists between the DDL statement and the first data node that previously executed the DDL statement. If it is determined that the DDL statement has connection information, the connection information is deleted first, and then the table information of the target database corresponding to the DDL statement is determined. For example, Figure 6 is a flowchart illustrating an exception handling mechanism provided in this application. As shown in Figure 6, if the maximum number of retries has not been exceeded, the task re-enters the distribution processing flow of the DDL execution module in a loop. During the retry process, it checks whether the previous subtask still exists; if it does, the connection is killed.
[0159] In one possible implementation, after step 305, the method further includes: if it is determined that the number of retries for the DDL statement is greater than a preset number, then the first execution result of the DDL statement is sent to the application device.
[0160] For example, referring to Figure 6, if the device determines that the number of retries for the currently retried DDL statement is greater than the preset number, it deletes the execution task and sends the execution result of the failed DDL statement, i.e., the first execution result, to the application device.
[0161] 307. Based on the metadata node, determine the second data node corresponding to the DDL statement according to the table information of the target database; wherein, the second data node is any first data node other than the first data node that executes the DDL statement.
[0162] For example, based on the metadata nodes deployed on the device, a table structure consistency check is performed according to the table information of the target database. According to the check result, if the table structure consistency passes, the first data nodes corresponding to the execution of the DDL statement are first determined, and then the second data nodes are determined from these first data nodes. That is, the second data node is any other first data node besides the first data node that successfully executed the DDL statement. For example, for DDL statement 1, data node 1, data node 2, and data node 3 all execute DDL statement 1. If it is determined that data node 1 and data node 2 successfully executed DDL statement 1 in the previous execution task, but data node 3 failed to execute DDL statement 1, then data node 3 is determined as the second data node to execute DDL statement 1 again.
[0163] In one possible implementation, 307 includes:
[0164] The first step is to identify the table information and obtain the identification result; the identification result represents the version status of the current table information.
[0165] The second step is to determine the second data node if the identification result representation table information is not the latest version.
[0166] Specifically, referring to Figure 6, based on the metadata node deployed on the device, a table structure consistency check is performed according to the table information of the target database corresponding to the current DDL statement. That is, the table information is identified and processed to obtain the identification result, which represents the current version of the table information. Then, based on the identification result, if it is determined that the table information is not the latest version, the second data node corresponding to the DDL statement is determined.
[0167] In one possible implementation, the table information includes field information, which represents the field type and length of the table structure; the first step in 307 includes: performing identification processing on the field information in the table information to obtain the identification result.
[0168] Specifically, each table information includes field information, and each field information represents the field type and length of the table structure; based on the metadata nodes deployed on the device, the field information in the table information of the target database corresponding to the current DDL statement is identified and processed to obtain the identification result corresponding to the table information.
[0169] In one possible implementation, after performing the first step in 306, 306 further includes:
[0170] The third step is to determine that if the identification result representation table information is the latest version, then determine the first execution result of the DDL statement to indicate that the DDL statement was executed successfully.
[0171] The fourth step is to send the first execution result of the DDL statement to the application device.
[0172] For example, referring to Figure 6, based on the metadata node deployed on the device, a table structure consistency check is performed according to the table information of the target database corresponding to the current DDL statement. That is, the table information is identified and processed to obtain the identification result, which represents the current version of the table information. If the identification result determines that the table information is the latest version, it means that the execution has been successful. Then, the first execution result of the current DDL statement is determined to be successful, the subtask table status is changed to success, and the retry process is skipped. The final result of the DDL statement, i.e. the first execution result, is returned to the Server layer. The Server layer replies to the Proxy, and then returns the result to the application device.
[0173] 308. Send the DDL statement to the second data node corresponding to the DDL statement; wherein the second data node is used to re-execute the DDL statement to obtain a second execution result.
[0174] For example, based on the metadata nodes deployed on the device, the second data node corresponding to the DDL statement is determined, the DDL statement is sent to the corresponding data node, and the DDL statement is re-executed to obtain the second execution result of the DDL statement, thus determining whether the DDL statement was executed successfully; thereby, automatic retries can be performed, simplifying the process and improving fault tolerance.
[0175] 309. Update the execution status of the DDL statement based on the second execution result of the DDL statement.
[0176] For example, after a retry, the device analyzes the retry result of the current DDL statement, i.e., the second execution result, to determine whether the retry execution of the current DDL statement was successful, and updates the execution status of the DDL statement accordingly.
[0177] In one possible implementation, 308 includes:
[0178] The first step is to determine if the second execution result indicates that the DDL statement was executed successfully, and then update the execution status of the DDL statement to a successful status.
[0179] The second step is to determine the number of retries for the DDL statement if the second execution result indicates that the DDL statement has failed. Then, the execution status of the DDL statement is updated according to the number of retries. The number of retries represents the total number of times the DDL statement has been re-executed.
[0180] For example, referring to Figure 6, for each retried DDL statement, the device analyzes the retry result of each DDL statement. If it is determined that the DDL statement was executed successfully, the execution status of the DDL statement is updated to a successful status, that is, the status in the subtask table can be updated to a successful status. If it is determined that the DDL statement was executed unsuccessfully, the number of retries for the DDL statement is determined again, and the execution status of the DDL statement is updated according to the number of retries.
[0181] In one possible implementation, the second step in 308 includes:
[0182] Step 1: If it is determined that the number of retries for the DDL statement is greater than the preset number, then update the execution status of the DDL statement to a failed state and generate a prompt message; the prompt message indicates that the current DDL statement has failed to execute, and the prompt message is used to prompt manual handling.
[0183] Step 2: If the number of retries for the DDL statement is less than or equal to the preset number, update the execution status of the DDL statement to "in execution"; and proceed with the step of determining the table information of the target database.
[0184] Specifically, referring to Figure 6, if the device determines that the number of retries for a currently failed DDL statement exceeds the preset number (i.e., it has exceeded the maximum number of retries and still has not succeeded), then the execution status of the DDL statement is updated to a failed state. A prompt message is generated to indicate that the current DDL statement has failed, and this prompt message is used to suggest manual intervention, i.e., human intervention is required to investigate the cause of the blocked DDL, such as insufficient disk space. After the problem is resolved, the DDL operation can be directly redone. Data nodes that have already successfully executed will skip this subtask, ultimately ensuring the consistency of the table structure of all data nodes after the DDL succeeds. If the device determines that the number of retries for a currently failed DDL statement is less than or equal to the preset number (i.e., it has not exceeded the maximum number of retries), then the execution status of the DDL statement is updated to an in-process state; and the distribution process is re-executed, i.e., the step of determining the table information of the target database is performed.
[0185] 310. If it is determined that the execution status of each DDL statement has been updated and processed, then the second execution result of the DDL statement is sent to the application device.
[0186] For example, for each retried DDL statement, if it is determined that the execution status of each DDL statement has been updated and processed, that is, all DDL statements have been executed and their status has been updated, then the final execution result of these DDL statements is sent to the application device.
[0187] The database processing method provided in this application incorporates a DDL scheduling module and an execution module within the database kernel for scheduling DDL jobs and constructing DDL tasks. This enables collaborative implementation of single-point DDL in a distributed database without the need for third-party components, avoiding the complex operations of directly connecting to each node for execution. Furthermore, because the DDL module reuses the database kernel's syntax parsing capabilities, syntax compatibility is significantly improved.
[0188] The database processing method provided in this application embodiment, on the one hand, designs a DDL scheduling module and an execution module in the database kernel to schedule DDL jobs and construct DDL tasks, so as to collaboratively realize single-point DDL of distributed database without introducing third-party components and avoid the complex operation of directly connecting to each node for execution; on the other hand, the distributed database DDL fault tolerance mechanism automatically identifies whether the table structure is the latest version when the task fails, and can automatically retry according to the result, thereby improving fault tolerance and improving database processing efficiency.
[0189] Figure 7 is a schematic diagram of a database processing device provided in this application. As shown in Figure 7, the database processing device provided in this embodiment includes:
[0190] The first execution unit 401 is configured to receive a Data Definition Language (DDL) request sent by the application device, and execute each DDL statement in the DDL request based on at least one first data node to obtain the first execution result of each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents database table operations used to process each database.
[0191] The determining unit 402 is used to update the execution status of the DDL statement to a failure status if the first execution result indicates that the DDL statement has failed to execute; and to determine the number of retries for the DDL statement; wherein the number of retries indicates the total number of times the DDL statement has been re-executed.
[0192] The second execution unit 403 is used to determine the table information of the target database if the number of retries for the DDL statement is less than or equal to a preset number, and to execute the DDL statement again based on the table information of the target database to obtain the second execution result of the DDL statement; wherein, the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database.
[0193] The sending unit 404 is used to send the second execution result of the DDL statement to the application device.
[0194] In one possible implementation, the second execution unit 403 is specifically configured to: determine the second data node corresponding to the DDL statement based on the metadata node and the table information of the target database; wherein the second data node is any first data node other than the first data node where the DDL statement was successfully executed; and send the DDL statement to the second data node corresponding to the DDL statement; wherein the second data node is used to re-execute the DDL statement to obtain a second execution result.
[0195] In one possible implementation, the second execution unit 403 is specifically used to: perform identification processing on the table information to obtain an identification result; wherein the identification result represents the version status of the current table information; if it is determined that the identification result represents that the table information is not the latest version, then a second data node is determined.
[0196] In one possible implementation, the table information includes field information, which represents the field type and length of the table structure; the second execution unit 403 is specifically used to: perform recognition processing on the field information in the table information to obtain recognition results.
[0197] In one possible implementation, the second execution unit 403 is further specifically configured to: if it is determined that the identification result representation table information is the latest version, determine that the first execution result of the DDL statement represents that the DDL statement was executed successfully; and send the first execution result of the DDL statement to the application device.
[0198] In one possible implementation, the first execution unit 401 is specifically used to: generate a DDL task corresponding to a DDL request based on a metadata node; wherein the DDL task includes at least one DDL statement in the DDL request; the DDL task has task information; the task information represents the execution status of each DDL statement in the DDL task; write the task information of each DDL task into a subtask table; and execute the DDL statement in the DDL task based on each first data node to obtain the first execution result of the DDL statement.
[0199] In one possible implementation, the first execution unit 401 is specifically configured to: encapsulate and process each DDL statement in the DDL request based on the metadata node to obtain the DDL job corresponding to the DDL request; wherein, the DDL job has job information; the job information is the database table information to be processed corresponding to the DDL job; if it is determined that the DDL job is allowed to be executed, the job information is stored in the metadata table; wherein, the metadata table includes a job table and an information table; the job table is used to record the execution status of the DDL statements that are currently being executed or not executed; the information table is used to record the changes in the target database before and after the execution of the DDL statement; determine the number of nodes, and generate each DDL task according to the number of nodes; wherein, the number of nodes is the total number of first data nodes that can be used to execute database operations; the DDL task corresponds one-to-one with the first data node.
[0200] In one possible implementation, the first execution unit 401 is specifically used to: if it is determined that there is no target DDL job in the job table, then determine that the DDL job is allowed to be executed; wherein, the target DDL job is a DDL job whose job information is consistent with the job information of the DDL job.
[0201] In one possible implementation, the information table includes table change information and each DDL statement in the DDL job; the table change information represents the changes in the table structure of the target database before and after the execution of the DDL statements.
[0202] In one possible implementation, the task information includes node information, database information, and log information; the node information is the data node information corresponding to the execution of the DDL task; the database information is the connection identifier information of the target database corresponding to the DDL task; and the log information is the error information generated during the execution of the DDL task.
[0203] In one possible implementation, the sending unit 404 is specifically used to: update the execution status of the DDL statement according to the second execution result of the DDL statement; if it is determined that the execution status of each DDL statement has been updated, then send the second execution result of the DDL statement to the application device.
[0204] In one possible implementation, the sending unit 404 is specifically configured to: if it is determined that the second execution result indicates that the DDL statement was executed successfully, then update the execution status of the DDL statement to a successful status; if it is determined that the second execution result indicates that the DDL statement was executed unsuccessfully, then determine the number of retries for the DDL statement; and update the execution status of the DDL statement according to the number of retries for the DDL statement; wherein the number of retries indicates the total number of times the DDL statement is re-executed.
[0205] In one possible implementation, the sending unit 404 is specifically configured to: if it is determined that the number of retries for the DDL statement is greater than a preset number, update the execution status of the DDL statement to a failed state; and generate a prompt message; wherein the prompt message indicates that the current DDL statement has failed to execute, and the prompt message is used to prompt manual processing; if it is determined that the number of retries for the DDL statement is less than or equal to the preset number, update the execution status of the DDL statement to an in-process state; and perform the step of determining the table information of the target database.
[0206] In one possible implementation, before the second execution unit 403 is specifically used to determine the table information of the target database, the apparatus is further configured to: if it is determined that the DDL statement has connection information, delete the connection information of the DDL statement; wherein the connection information represents the connection relationship between the DDL statement and the first data node that last executed the DDL statement.
[0207] In one possible implementation, the apparatus is further configured to: if it is determined that the first execution result indicates that the DDL statement was executed successfully, update the execution status of the DDL statement to a successful status; and send the first execution result of the DDL statement to the application device.
[0208] In one possible implementation, the apparatus further includes: if it is determined that the number of retries for the DDL statement is greater than a preset number, then sending the first execution result of the DDL statement to the application device.
[0209] The device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0210] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 8, the electronic device includes: a memory 501 and a processor 502; the memory 501 is a memory for storing executable instructions of the processor 502.
[0211] The processor 502 is configured to perform the method provided in the above embodiments.
[0212] The electronic device 500 also includes a receiver 503 and a transmitter 504. The receiver 503 is used to receive instructions and data sent by other devices, and the transmitter 504 is used to send instructions and data to external devices.
[0213] The specific implementation process of processor 502 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0214] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0215] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0216] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0217] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0218] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0219] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0220] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0221] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0223] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0224] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0225] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0226] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A database processing method, characterized in that, include: The system receives a Data Definition Language (DDL) request sent by an application device, and executes each DDL statement in the DDL request based on at least one first data node to obtain a first execution result for each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents database table operations used to process each database. If the first execution result indicates that the DDL statement has failed, then the execution status of the DDL statement is updated to a failed state; and the number of retries for the DDL statement is determined; wherein, the number of retries represents the total number of times the DDL statement is re-executed; If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the table information of the target database is determined, and the DDL statement is executed again based on the table information of the target database to obtain the second execution result of the DDL statement; wherein, the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database. The second execution result of the DDL statement is sent to the application device.
2. The method according to claim 1, characterized in that, Based on the table information of the target database, the DDL statement is executed again to obtain a second execution result of the DDL statement, including: Based on the metadata node, and according to the table information of the target database, the second data node corresponding to the DDL statement is determined; wherein, the second data node is a first data node other than the first data node that successfully executed the DDL statement; The DDL statement is sent to the second data node corresponding to the DDL statement; wherein the second data node is used to re-execute the DDL statement to obtain the second execution result.
3. The method according to claim 2, characterized in that, Based on the metadata node, and according to the table information of the target database, the second data node corresponding to the DDL statement is determined, including: The table information is processed to obtain an identification result; wherein, the identification result represents the current version status of the table information; If it is determined that the identification result indicates that the table information is not the latest version, then the second data node is determined.
4. The method according to claim 3, characterized in that, The table information includes field information, which characterizes the field types and lengths of the table structure. The table information is processed to obtain the recognition result, including: The field information in the table is identified to obtain the identification result.
5. The method according to claim 3, characterized in that, After processing the table information to obtain the recognition result, the method further includes: If the identification result indicates that the table information is the latest version, then the first execution result of the DDL statement indicates that the DDL statement was executed successfully. The first execution result of the DDL statement is sent to the application device.
6. The method according to claim 1, characterized in that, Based on at least one first data node, each DDL statement in the DDL request is executed to obtain a first execution result for each DDL statement, including: Based on the metadata node, a DDL task corresponding to the DDL request is generated; wherein, the DDL task includes at least one DDL statement from the DDL request; the DDL task has task information; the task information represents the execution status of each DDL statement in the DDL task; Write the task information of each DDL task into the subtask table; Based on each of the first data nodes, the DDL statements in the DDL task are executed to obtain the first execution result of the DDL statements.
7. The method according to claim 6, characterized in that, Based on the metadata node, generate the DDL task corresponding to the DDL request, including: Based on the metadata node, each DDL statement in the DDL request is encapsulated to obtain the DDL job corresponding to the DDL request; wherein, the DDL job has job information; the job information is the database table information to be processed corresponding to the DDL job; If it is determined that the DDL job is allowed to be executed, the job information is stored in a metadata table; wherein, the metadata table includes a job table and an information table; the job table is used to record the execution status of currently executing or unexecuted DDL statements; the information table is used to record the changes in the target database before and after the execution of the DDL statement; The number of nodes is determined, and each DDL task is generated based on the number of nodes; wherein the number of nodes is the total number of first data nodes that can be used to perform database operations; and each DDL task corresponds one-to-one with the first data node.
8. The method according to claim 7, characterized in that, Determining that the DDL job is allowed to be executed includes: If it is determined that the target DDL job does not exist in the job table, then the DDL job is determined to be allowed to be executed; wherein, the target DDL job is a DDL job whose job information is consistent with the job information of the DDL job.
9. The method according to claim 7, characterized in that, The information table includes table change information and each DDL statement in the DDL job; the table change information represents the changes in the table structure of the target database before and after the execution of the DDL statements.
10. The method according to claim 6, characterized in that, The task information includes node information, database information, and log information; the node information is the data node information corresponding to the execution of the DDL task; the database information is the connection identifier information of the target database corresponding to the DDL task; and the log information is the error information generated during the execution of the DDL task.
11. The method according to claim 1, characterized in that, Sending the second execution result of the DDL statement to the application device includes: Based on the second execution result of the DDL statement, the execution status of the DDL statement is updated. If it is determined that the execution status of each DDL statement has been updated and processed, the second execution result of the DDL statement is sent to the application device.
12. The method according to claim 11, characterized in that, Based on the second execution result of the DDL statement, the execution status of the DDL statement is updated, including: If it is determined that the second execution result indicates that the DDL statement was executed successfully, then the execution status of the DDL statement is updated to a successful status; If the second execution result indicates that the DDL statement has failed to execute, then the number of retries for the DDL statement is determined; and the execution status of the DDL statement is updated according to the number of retries for the DDL statement; wherein the number of retries represents the total number of times the DDL statement has been re-executed.
13. The method according to claim 12, characterized in that, Update the execution status of the DDL statement based on the number of retries, including: If it is determined that the number of retries for the DDL statement is greater than the preset number, the execution status of the DDL statement is updated to a failed state; and a prompt message is generated; wherein, the prompt message indicates that the current execution of the DDL statement has failed, and the prompt message is used to prompt manual handling; If it is determined that the number of retries for the DDL statement is less than or equal to the preset number, then the execution status of the DDL statement is updated to the in-process status; and the step of determining the table information of the target database is executed.
14. The method according to claim 1, characterized in that, Before determining the table information of the target database, the following is also included: If it is determined that the DDL statement has connection information, then the connection information of the DDL statement is deleted; wherein, the connection information represents the connection relationship between the DDL statement and the first data node that last executed the DDL statement.
15. The method according to any one of claims 1-14, characterized in that, The method further includes: If the first execution result indicates that the DDL statement was executed successfully, the execution status of the DDL statement is updated to a successful status; and the first execution result of the DDL statement is sent to the application device.
16. The method according to any one of claims 1-14, characterized in that, The method further includes: If it is determined that the number of retries for the DDL statement is greater than the preset number, then the first execution result of the DDL statement is sent to the application device.
17. A database processing apparatus, characterized in that, include: The first execution unit is configured to receive a Data Definition Language (DDL) request sent by an application device, and execute each DDL statement in the DDL request based on at least one first data node to obtain a first execution result for each DDL statement; wherein, the DDL request is used to request processing of at least one database; the database is used to store financial business data; and the DDL statement represents database table operations used to process each database. The determining unit is configured to update the execution status of the DDL statement to a failed state if the first execution result indicates that the DDL statement has failed to execute; and to determine the number of retries for the DDL statement; wherein the number of retries indicates the total number of times the DDL statement has been re-executed. The second execution unit is configured to, if it is determined that the number of retries for the DDL statement is less than or equal to a preset number, determine the table information of the target database, and execute the DDL statement again based on the table information of the target database to obtain the second execution result of the DDL statement; wherein, the target database is the database to be processed corresponding to the DDL statement; and the table information is the table structure information of the target database. The sending unit is used to send the second execution result of the DDL statement to the application device.
18. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-16.
19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-16.
20. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-16.
Citation Information
Patent Citations
Method and device for realizing data definition operation atomicity of distributed database
CN117435600A
Data redistribution initiation method and device, electronic equipment and storage medium
CN117762967A
Webpage creation method and device, server and storage medium
CN117850779A
Database processing method, electronic equipment, storage medium and program product
CN118445303A
Improved execution of operations on partitioned tables
US20230385263A1
Cited By
Data migration method for black box big data service system
CN121455929A