DDL task parallel processing method, computing node, and electronic device

Abstract

Embodiments of the present application provide a parallel processing method of DDL task, a computing node and an electronic device, and relate to the field of databases. The method comprises: obtaining a job task table, the job task table being used for recording related information of an unprocessed completed DDL task, the related information comprising an order in which the DDL task enters a database system and a data mode in which a change object corresponding to the DDL task is located; determining a first DDL task being executed, and determining a second DDL task meeting a pre-determined task parallel processing rule from the job task table; and processing the target second DDL task and the first DDL task in parallel. Embodiments of the present application have simple logic and strong scalability, can adapt to deployment of different data block forms, and do not have the limitation that the prior art relies on a metadata lock mode to coordinate execution order to achieve concurrent DDL scheduling effect.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of databases, and in particular relates to a parallel processing method for DDL tasks, a computing node, an electronic device and a computer readable storage medium. BACKGROUND

[0002] At present, distributed scheduling systems of data definition language (DDL) statements mainly include the following centralized implementations:

[0003] 1. A traditional single machine or cluster system taking Oracle, MySQL and other traditional databases as examples. A Meta data Lock system is usually implemented to coordinate the execution sequence of various DDL statements by locking the objects that need to be changed, so as to achieve concurrency.

[0004] 2. A distributed database system Ocean Base and a database of TDSQL also achieve the effect of concurrent DDL statement scheduling by implementing a kind of MDL lock in the distributed system.

[0005] How to ensure that multiple DDL workers (also referred to as task execution units) do not cause inconsistency of object definitions or object storage data in a user database when concurrently executing DDL tasks is a problem to be solved. SUMMARY

[0006] The embodiments of the present application provide a parallel processing method for DDL tasks, a computing node, an electronic device and a computer readable storage medium, which can solve the above problems in the prior art. The technical solution is as follows:

[0007] According to an aspect of the embodiments of the present application, a parallel processing method for data definition language (DDL) tasks is provided, which is executed by a target computing node in a database system, and the method comprises the following steps:

[0008] obtaining a job task table, wherein the job task table is used to record related information of unprocessed DDL tasks, and the related information comprises an order of the DDL tasks entering the database system and a data mode of a change object corresponding to the DDL tasks;

[0009] determining a first DDL task being executed, determining a second DDL task meeting a predetermined task parallel processing rule from the job task table, and the task parallel processing rule is related to the order of the DDL tasks entering the database system and the data mode of the change object corresponding to the DDL tasks;

[0010] The target second DDL task is processed in parallel with the first DDL task.

[0011] According to another aspect of the embodiments of the present application, a target computing node in a database system is provided, which comprises:

[0012] a job task table module configured to obtain a job task table, the job task table being configured to record relevant information of unprocessed completed DDL tasks, the relevant information comprising an order in which the DDL tasks enter the database system and a data mode in which a change object corresponding to the DDL task is located;

[0013] a task determination module configured to determine a first DDL task being executed, and determine a second DDL task meeting a predetermined task parallel processing rule from the job task table, the task parallel processing rule being related to the order in which the DDL tasks enter the database system and the data mode in which the change object corresponding to the DDL task is located;

[0014] a parallel processing module configured to process the second DDL task in parallel with the first DDL task.

[0015] According to another aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the parallel processing method of the DDL task.

[0016] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, the computer program is executed by a processor to implement the steps of the parallel processing method of the DDL task.

[0017] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0018] By obtaining the job task table, the relevant information of at least one DDL task is recorded in the job task table, based on the task parallel processing rule, the second DDL task which can be processed in parallel with the first DDL task being processed is determined from the job task table, and the second DDL task is processed in parallel with the first DDL task, which has simple logic, strong scalability, can adapt to the deployment of different data block forms, and does not have the limitation that the DDL task can only be selected from the queue according to the order in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0020] Figure 1A flowchart of an execution process of a DDL task provided by the related art is shown in FIG. 1.

[0021] Figure 2 A flowchart of a parallel processing method of a DDL task provided by an embodiment of the present application is shown in FIG. 2.

[0022] Figure 3 A tree structure composed of three levels of change objects provided by an embodiment of the present application is shown in FIG. 3.

[0023] Figure 4 A flowchart of a processing method of a DDL task provided by another embodiment of the present application is shown in FIG. 4.

[0024] Figure 5 A flowchart of a processing method of a DDL task provided by another embodiment of the present application is shown in FIG. 5.

[0025] Figure 6 A structure diagram of a target computing node provided by an embodiment of the present application is shown in FIG. 6.

[0026] Figure 7 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0027] Embodiments of the present application will be described herein below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0028] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein can also include the plural forms. It should be further understood that the terms "comprise" and "include" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or can mean that the element and the other element are connected through an intermediate element. In addition, the "connection" or "coupling" used herein can include wireless connection or wireless coupling. The term "and / or" used herein means that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0029] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0030] The solution currently adopted by the implementation of the common DDL statement is that all the DDL tasks in the distributed cluster are registered in a queue in the order of entering the cluster, and the registration in the queue is mainly for the purpose of fault recovery. The tasks are acquired and executed by a task execution unit for executing the DDL tasks. The specific execution process is shown in Figure 1 As shown in the figure, the computing nodes can accept the DDL change statement from the client, and then store the statement in the DDL task queue. The task execution unit takes one DDL task from the DDL task queue each time for execution. Since the execution order of the DDL task is only related to the sorting of the DDL task in the DDL task queue, when multiple task execution units execute the DDL task concurrently, the consistency of the object definition or the object storage data in the user database may be caused. The database usually uses MDL lock to ensure the correct execution between the DDL statements.

[0031] The DDL task parallel processing, device, electronic equipment, computer readable storage medium and computer program product provided by the present application aim to solve the above technical problems of the prior art.

[0032] The technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be pointed out that the following embodiments can be mutually referenced, borrowed or combined, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0033] A DDL task parallel processing method is provided in the embodiments of the present application, as shown in Figure 2 The method comprises the following steps.

[0034] S101, obtaining a job task table, the job task table being used for recording the related information of the unprocessed completed DDL task, and the related information comprising the order of the DDL task entering the database system and the data schema of the change object corresponding to the DDL task.

[0035] The client can obtain at least one DDL task according to the DDL statement after sending the DDL statement to the database system. The embodiments of the present application can set the task identifier of each DDL task according to the order of the DDL task entering the database system, and write the related information of the DDL task into the job task table.

[0036] The related information of the DDL task in the embodiment of the present application can include a task identifier of the DDL task, meta information of the DDL task (i.e. key information for executing the DDL task to ensure correct execution of the DDL task and abnormality handling), a computing node for executing the DDL task, and the like, which are not specifically limited in the embodiment of the present application.

[0037] In one embodiment, the task identifier is used to represent the sequence of the DDL task entering the database system, for example, the task identifier of the first DDL task entering the database system is 1, the task identifier of the second DDL task entering the database system is 2, and so on.

[0038] S102, determining a first DDL task being executed, determining a second DDL task meeting a predetermined task parallel processing rule from the job task table, the task parallel processing rule being related to the sequence of the DDL task entering the database system and the data mode of the change object corresponding to the DDL task;

[0039] The embodiment of the present application records the related information of the DDL task to be processed in the job task table, and when determining the second DDL task being executed, there is no limitation that the DDL task can only be selected from the queue according to the sequence as in the prior art.

[0040] The DDL task concurrency rule in the embodiment of the present application includes but is not limited to the following contents:

[0041] 1) The DDL task corresponding to the DDL statement on the same data table needs to be executed according to the sequence of the DDL task entering the database system;

[0042] 2) The DDL task corresponding to the DDL statement on different data tables can be executed concurrently;

[0043] 3) The DDL task corresponding to the change object of different levels in the data also needs to be executed according to the sequence of the levels.

[0044] Please refer to Figure 3 which exemplarily shows a tree structure composed of three levels of change objects in the embodiment of the present application, wherein the root node of the tree structure is the change object of the first level: DB (schema), the change objects of the second level include: function, stored procedure, Table, and view, and for the Table, the change objects of the third level include: index, column, tigger and constraint.

[0045] In an embodiment, the embodiment of the present application implements a task parallel processing rule based on an SQL statement. The advantage of implementing the task parallel processing rule based on the SQL statement is that it is a lock-free and very lightweight operation, which does not affect the execution of DML and other DDL tasks.

[0046] The task parallel processing rule implemented by the embodiment of the present application based on the SQL statement includes:

[0047] Statement 1: select * from tidb_ddl_job where job_id in (select min(job_id) from tidb_ddl_job where not reorg group by table_id) and exec_owner=0;

[0048] The meaning of statement 1 is: find the current second DDL task. If it is a physical DDL task, after selecting to execute the second DDL task, the related information of the second DDL task needs to be recorded in the reorganization information table. exec_owner=0 indicates that the DDL task has not been executed.

[0049] Statement 2: select * from tidb_ddl_job where is_drop_schema and schema_id={job.SchemaID} and job_id<{job.id} limit 1;

[0050] The meaning of statement 2 is: if it is a second or third level change object, it is judged whether there is a DDL task corresponding to a change object of a higher level executed.

[0051] Statement 3: select * from mysql.tidb_ddl_job where schema_id={job.SchemaID} and job_id<{job.id} limit 1;

[0052] The meaning of statement 3 is: if it is a first level change object, it is judged whether there is a DDL task corresponding to a change object of a lower level executed.

[0053] It should be noted that the above three SQL statements are implemented for the TiDB open source distributed relational database. If other databases are implemented, the SQL statements will be changed, but the essence does not change. The embodiment of the present application will not be repeated.

[0054] S103, parallel processing the second DDL task and the first DDL task.

[0055] The second DDL task can be processed in parallel with the first DDL task being executed after the second DDL task is determined. The number of the first DDL tasks of the embodiments of the present application is not specifically limited, for example, can be one or multiple.

[0056] The embodiments of the present application determine the second DDL task that can be processed in parallel with the first DDL task being processed based on the task parallel processing rule by obtaining the job task table recording the related information of at least one unprocessed DDL task, and process the second DDL task in parallel with the first DDL task, which realizes simple logic and strong scalability, can adapt to the deployment of different data block forms, and does not have the limitation that the DDL task can only be selected from the queue according to the sequence in the prior art.

[0057] Based on the above embodiments, as an optional embodiment, the related information includes a computing node identifier for indicating a computing node currently executing the corresponding DDL task, and the job task table includes a first field for recording the computing node identifier.

[0058] Specifically, the embodiments of the present application can set an exec owner field (i.e. the first field) in the job task table, and record the computing node identifier of the computing node currently executing the corresponding DDL task in the field. It should be understood that when the DDL task is not executed, the first field can be empty or other marks (e.g. 0) indicating that it is not executed. If the DDL task is executed but not successfully, the field still records the computing node identifier of the computing node executing the DDL task.

[0059] Processing the second DDL task in parallel with the first DDL task includes:

[0060] S201, if it is determined according to the first field of the second DDL task in the job task table that the second DDL task is executed for the first time, recording the target computing node identifier of the target computing node in the first field;

[0061] S202, calling a task execution unit in the database system to execute the second DDL task, and the type of the task execution unit includes at least one of a process, a thread, and a coroutine.

[0062] The embodiments of the present application need to initialize the second DDL task when it is determined that the second DDL task is executed for the first time, and record the target computing node identifier of the target computing node in the first field, which is convenient for rollback if the second DDL task is not successfully executed.

[0063] After recording the target computing node identifier, this embodiment of the application can invoke a task execution unit to execute the second DDL task. It should be noted that this embodiment of the application has multiple task execution units, and the computing node can select an idle task execution unit from among these units to execute the second DDL task. Multiple task execution units can execute DDL tasks in parallel. In some embodiments, a DDL task can be split into multiple subtasks, with each task execution unit executing one subtask.

[0064] Based on the above embodiments, as an optional embodiment, the relevant information includes the object identifier of the object to be changed corresponding to the DDL task and the metadata of the DDL task.

[0065] Specifically, in this embodiment of the application, a table_id field (i.e., the second field) can be set in the job task table to record the object identifier of the changed object corresponding to the DDL task, a schema_id field (i.e., the third field) can be set to record the schema identifier of the changed object, and a job meta field (i.e., the fourth field) can be set to record the meta information of the DDL task.

[0066] The second DDL task is executed by calling the task execution unit in the database system, including:

[0067] S301. Call the task execution unit to read the second to fourth fields of the second DDL task in the job task table to obtain the target object identifier of the target change object corresponding to the second DDL task, the target mode identifier of the target data mode in which the target change object is located, and the target meta information of the second DDL task.

[0068] S302. Determine the target change object from the target data pattern of the database system based on the target object identifier and the target pattern identifier;

[0069] S303. Based on the target metadata, execute the second DDL task on the target change object.

[0070] The task execution unit in this embodiment can obtain the target object identifier of the target change object corresponding to the second DDL task, the target mode identifier of the target schema of the target change object, and the target meta-information of the second DDL task by reading the second to fourth fields of the second DDL task in the job task table. Based on the target object identifier and the target mode identifier, the target change object can be determined from the database system. Finally, the second DDL task is executed on the target change object based on the target meta-information, which can achieve accurate determination of the change object and task processing.

[0071] Based on the above embodiments, as an optional embodiment, parallel processing of the second DLL task further includes:

[0072] S401. If it is determined from the first field of the second DDL task in the job task table that the second DDL task is not being executed for the first time, then determine whether to resume the execution of the second DDL task.

[0073] S402. If it is determined that the execution of the second DDL task should be resumed, then the second DDL task is executed. If it is determined that the execution of the second DDL task should not be resumed, then the data generated in the last execution of the second DDL task is rolled back.

[0074] It should be noted that when the first field records the compute node identifier of compute nodes other than the target compute node, it means that the second DDL task is not being executed for the first time and the previous execution failed. In this case, it is necessary to determine whether to resume the execution of the second DDL task (if the DDL task has been completed, it will be archived in the historical task table; if it is still in the job task list, it means that it still needs to be executed).

[0075] If it is determined that the execution of the second DDL task should be resumed, then the second DDL task is executed. Specifically, in this embodiment, the progress of the previous execution of the second DDL task can be recorded in the job task table, so that the execution of the second DDL task can continue according to the progress of the previous execution. If it is determined that the execution of the second DDL task should not be resumed, then the data generated by the previous execution of the second DDL task is rolled back. It can be understood that after the rollback, the second DDL task will no longer be executed.

[0076] Based on the above embodiments, as an optional embodiment, the relevant information includes a type identifier for indicating whether the corresponding DDL task is a logical DDL task, and the job task table includes a fifth field for recording the type identifier;

[0077] DDL tasks can generally be divided into logical DDL and physical DDL. Logical DDL only requires modifying the definition of objects in the database (tables, indexes, columns, etc.) (e.g., changing table names). Physical DDL, on the other hand, usually involves a reorganization process. This process typically requires scanning the entire data table once to synchronize the data on the table to the newly added or modified objects, ensuring data consistency.

[0078] The job task table includes a reorg field (also known as the fifth field) for recording the type identifier.

[0079] Processing the second DDL task in parallel with the first DDL task includes:

[0080] S501. If it is determined that the second DDL task is not a logical DDL task based on the fifth field of the second DDL task in the job task table, then the second DDL task is split into multiple subtasks.

[0081] S502. Based on the number of subtasks, call the corresponding number of task execution units to execute the subtasks in parallel.

[0082] Specifically, in this embodiment of the application, if the second DDL task is determined to be a logical DDL task based on the fifth field of the second DDL task, then the second DDL task is not split, that is, only one task execution unit is called to execute the second DDL task. If the second DDL task is not a logical DDL task, that is, a physical DDL task, then the second DDL task is split into multiple subtasks, and the corresponding number of task execution units execute the subtasks in parallel.

[0083] Based on the above embodiments, as an optional embodiment, the process further includes calling a corresponding number of task execution units to execute subtasks, and prior to this:

[0084] Create the reorganization information table for the second DDL task.

[0085] Record the metadata of the subtask in the reorganization information table;

[0086] The step of invoking a corresponding number of task execution units to execute subtasks includes:

[0087] The task execution unit is invoked to execute the corresponding subtask based on the metadata of the corresponding subtask in the reorganization information table.

[0088] Specifically, the metadata of a subtask includes at least one of the following: the data location at which the subtask begins and ends scanning, the node identifier of the target computing node, and the execution status of the subtask.

[0089] The recombination information table in this embodiment includes the following fields:

[0090] Job id, used to record the task identifier of DDL tasks;

[0091] `reorg_obj_id` is used to record the object identifier of the changed object that needs to be reorganized, such as the identifier of `index` in `add index`.

[0092] physical_id requires the id from the scan table;

[0093] The `dist para` parameter is used to record metadata about each subtask for parallel scheduling.

[0094] In some embodiments, the dist para field includes:

[0095] `exec_dist` indicates whether the DDL task is executed in a distributed manner across the entire cluster.

[0096] is_canceled is used to indicate whether to cancel the execution of the DDL task;

[0097] curr_reorg_type is used to indicate the task type of reorg;

[0098] reorg sub-tasks, further including:

[0099] The sub-scan start point is used to record the data position at which the subtask begins scanning.

[0100] Sub-scan end point is used to record the data position at the end of the sub-task scan.

[0101] The sub-scan executor is used to record the computation nodes of subtasks;

[0102] The subtask status is used to record the execution status of subtasks.

[0103] In one alternative embodiment, the task parallel processing rules are implemented based on Structured Query Language (SQL) statements;

[0104] Determine a second DDL task from the job task table that conforms to a predetermined parallel processing rule, including:

[0105] Select a reference DDL task from the job task table, and write the task identifier and mode identifier of the reference DDL task into the task parallel processing rule. If the task parallel processing rule is determined to meet the preset conditions, then the reference DDL task is used as the target DDL task.

[0106] Based on the above embodiments, as an optional embodiment, the second DDL task and the first DDL task are processed in parallel, and then the process further includes:

[0107] After the second DDL task is completed, the relevant information of the second DDL task is moved from the job task table to the task history table, which includes the fields in the job task table.

[0108] Please see Figure 4 The figure illustrates a flowchart of a DDL task processing method according to another embodiment of this application, as shown in the figure, including:

[0109] Step S601: The target computing node starts running;

[0110] Step S602: The target computing node selects the second DDL task from the job task table according to the task parallel processing rules.

[0111] Step S603: Start the task execution node to concurrently execute the logic of steps S603-608, and determine whether the second DDL task is started for the first time. If not, proceed to step S604; if yes, proceed to step S605.

[0112] Step S604: Initialize the second DDL task, update the job task table, and then proceed to step S610.

[0113] Step S605: Determine whether to resume the second DDL task. If yes, proceed to step S606; otherwise, proceed to step S607.

[0114] Step S606: Restore the previously saved execution progress of the second DDL task and continue execution. After the second DDL task is completed or an abnormal situation occurs, proceed to step 608.

[0115] Step S607: Roll back the data generated during the previous execution of the second DDL task;

[0116] Step S608: End the execution of the second DDL task and move the relevant information of the second DDL task from the job task table to the task history table. The task history table includes the fields in the job task table.

[0117] Step S609: The compute node initiates a background task in parallel, continuously selecting executable DDL tasks. If there are selectable DDL tasks, proceed to step S602; otherwise, proceed to step S610.

[0118] Step S610: Determine whether the computation node has finished running. If not, proceed to step S611; if yes, proceed to step S612.

[0119] Step S611: Sleep for one cycle, for example, 1 second, and then proceed to step SS609;

[0120] Step S612, process ends.

[0121] Please see Figure 5 The figure illustrates a flowchart of a DDL task processing method according to another embodiment of the present application, as shown in the figure, including:

[0122] S701. Each physics DDL task has a master-subtask, and the first subtask is used as the master-subtask.

[0123] S702. Determine whether a DDL task needs to be started. If yes, proceed to step S703; otherwise, proceed to step S706.

[0124] S703, Start the DDL task;

[0125] S704. Read the prefer_rule field of the DDL task in the job task table. The prefer_rule sets the parallel distribution strategy for the DDL task when it is executed in the cluster (e.g., setting the number of different task execution nodes in parallel according to the computing resources of different computing stages, or setting different subtasks on different computing nodes according to the data location). A simple distribution strategy can include setting all computing nodes in parallel in the same way.

[0126] S705. Divide the data to be backfilled into sub-tasks based on the number of sub-tasks generated by the DDL task, and reduce the number of sub-tasks accordingly. If the amount of data to be backfilled is small, reduce the number of sub-tasks accordingly. Initialize dist_para in the reorganization information table, initialize dist_para for each compute node, set exec_owner, and save the dist_para information.

[0127] S706. Each computing node obtains the currently active executable DDL tasks according to the task parallel processing rules, checks whether there are any subtasks that it needs to execute, and if so, restores the context of the subtask.

[0128] S707. Execute subtasks. In this embodiment of the application, the computing node starts the corresponding task execution node to execute the DDL task according to its own situation.

[0129] S708. Each computing node starts a task execution node and checks whether the execution of the DDL task has been cancelled. If yes, proceed to step S712; otherwise, proceed to step S709.

[0130] S709, Complete data processing;

[0131] S710. Update the subtask status in the reorganized information table. Here, an optimistic transaction mode can be used because if other subtasks are updating the same DDL task, the update of the subtask may fail. Therefore, it is only necessary to re-fetch the latest DDL task, update only the subtasks related to itself, and commit again until the update is successful. Alternatively, a pessimistic transaction mode can be used. Each time the subtask status needs to be committed, the latest DDL task is first obtained, the record is locked, and the transaction is committed after the update is completed.

[0132] S711. Determine whether the subtask is completed. If not, proceed to step S708. If yes, proceed to step S715.

[0133] S712. Determine if it is a master task. If not, proceed to step S715. If yes, proceed to step S713.

[0134] S713. Determine if all subtasks have ended. Specifically, check the status field of each subtask to determine if all subtasks have ended. If not, proceed to step S714; if yes, proceed to step S716.

[0135] S714, hibernate for a period of time;

[0136] S715, Subtask complete.

[0137] S716, DDL task completed.

[0138] It's important to note that when you need to cancel a DDL task, simply set the `is_canceled` field to `true`. All tasks, when updating subtask status, first need to read the task record and retrieve this field to determine whether to cancel the task. This also applies if a subtask exits abnormally; setting this field can stop the DDL task execution and return an error message based on the subtask status.

[0139] In the event of a compute node crash and restart, each task execution node on the compute node selects an inactive DDL task, checks its own subtasks, and restores them. Compute nodes can be numbered; that is, even if a compute node restarts on another machine, as long as the number remains the same, the subtasks can still be retrieved based on the number, restored, and execution resumed.

[0140] For compute nodes that crash and cannot start for an extended period, a mechanism is added to the compute nodes to detect changes in the cluster. If a compute node fails to rejoin the cluster for a period of time, the exec_owner of the subtask in the reorganization information table is changed to another node in the cluster or cleared to 0 (all active nodes can obtain the DDL task and resume execution), waiting for the task execution nodes of other compute nodes to resume execution.

[0141] This application has the following technical advantages:

[0142] 1. The solution has simple implementation logic and strong scalability.

[0143] 2. The solution can fully utilize the overall resources of the cluster, automatically balance the load of tasks, and has strong online scalability.

[0144] 3. It can adapt to deployments of different database formats;

[0145] a) Single-instance database deployment;

[0146] b) Single master node in the cluster, distributed DDL task scheduling across the entire cluster;

[0147] c) Cluster-distributed DDL task scheduling (multi-master node);

[0148] 4. It performs exceptionally well in fault recovery in distributed systems, unlike lock implementations which suffer from the complexity of distributed deadlock detection or the performance and single-point-of-failure issues of single-point lock control.

[0149] 5. Fault recovery is simple and efficient. This solution can restore DDL task scheduling to normal with virtually no additional operation required.

[0150] 6. It is easy to implement specific distributed scheduling rules in this model to schedule specific DDL tasks to specific computing nodes for execution, such as scheduling based on data location, availability of computing nodes in a multi-tenant environment, etc.

[0151] This application provides a target computing node in a database system, such as... Figure 6 As shown, this node may include: a job task table module 601, a task determination module 602, and a parallel processing module 603, wherein,

[0152] The job task table module 601 is used to obtain a job task table, which is used to record relevant information of unprocessed DDL tasks. The relevant information includes the order in which the DDL tasks enter the database system and the data mode of the change object corresponding to the DDL task.

[0153] The task determination module 602 is used to determine the first DDL task being executed and to determine the second DDL task that conforms to the pre-determined parallel processing rules from the job task table. The parallel processing rules are related to the order in which the DDL tasks enter the database system and the data mode of the change object corresponding to the DDL task.

[0154] Parallel processing module 603 is used to process the second DDL task and the first DDL task in parallel.

[0155] The nodes in this application embodiment can execute the methods provided in this application embodiment, and their implementation principles are similar. The actions performed by each module in the nodes of each embodiment of this application correspond to the steps in the methods of each embodiment of this application. For detailed functional descriptions of each module of the node, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0156] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a parallel processing method for DDL tasks. Compared with related technologies, this method can achieve the following: by obtaining a task job table, which records relevant information of at least one DDL task, implementing parallel processing rules based on SQL statements, determining a second DDL task that can be processed in parallel with other DDL tasks in the task job table, and processing the second DDL task in parallel with the first DDL task. The implementation logic is simple, and it has strong scalability, can adapt to deployments of different data block forms, and does not have the limitation of prior art that DDL tasks can only be selected from the queue in sorted order.

[0157] In one alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0158] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0159] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0160] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0161] The memory 4003 is used to store computer programs that execute the embodiments of this application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0162] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0163] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0164] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0165] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0166] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A parallel processing method for Data Definition Language (DDL) tasks, characterized in that, The method is executed by a target computing node in a database system, and comprises: obtaining a job task table, the job task table being used to record relevant information of an unprocessed completed DDL task, the relevant information comprising an order in which the DDL task enters the database system and a data mode in which a change object corresponding to the DDL task is located; determining a first DDL task being executed, determining a second DDL task from the job task table, the second DDL task meeting a predetermined task parallel processing rule, the task parallel processing rule being related to the order in which the DDL task enters the database system and the data mode in which the change object corresponding to the DDL task is located; the task parallel processing rule being implemented based on a select type SQL statement; parallel processing the second DDL task and the first DDL task; the task parallel processing rule comprises: selecting a task that does not perform data reorganization and is submitted earliest from unprocessed DDL tasks as a DDL task to be executed in parallel.

2. The method of claim 1, wherein, the relevant information comprises a computing node identifier used to indicate a computing node currently executing a corresponding DDL task, and the job task table comprises a first field used to record the computing node identifier; the parallel processing of the second DDL task and the first DDL task comprises: if it is determined according to the first field of the second DDL task in the job task table that the second DDL task is executed for the first time, recording a target computing node identifier of a target computing node in the first field; calling a task execution unit in the database system to execute the second DDL task, the type of the task execution unit comprising at least one of a process, a thread and a coroutine.

3. The method of claim 2, wherein, the relevant information comprises an object identifier of a change object corresponding to the DDL task and meta information of the DDL task; the job task table comprises a second field used to record the object identifier, a third field used to record a mode identifier of a data mode in which the change object corresponding to the DDL task is located and a fourth field used to record the meta information; the calling of the task execution unit to execute the second DDL task comprises: the calling of the task execution unit to read the second field to the fourth field of the second DDL task in the job task table, to obtain a target object identifier of a target change object corresponding to the second DDL task, a target mode identifier of a target data mode in which the target change object is located and target meta information of the second DDL task; determining the target change object from the target data mode of the database system according to the target object identifier and the target mode identifier; executing the second DDL task on the target change object according to the target meta information.

4. The method of claim 2, wherein, the parallel processing of the second DDL task further comprises: if it is determined according to the first field of the second DDL task in the job task table that the second DDL task is executed for the second time, judging whether to resume execution of the second DDL task. If it is determined to resume the execution of the second DDL task, the second DDL task is continued to be executed according to the progress of the last execution of the second DDL task recorded in the job task table record, and if it is determined not to resume the execution of the second DDL task, data generated by the last execution of the second DDL task is rolled back.

5. The method of claim 1, wherein, The related information includes a type identifier used to indicate whether the corresponding DDL task is a logical DDL task, and the job task table includes a fifth field used to record the type identifier; The parallel processing of the second DDL task and the first DDL task includes: If it is determined that the second DDL task is not a logical DDL task according to the fifth field of the second DDL task in the job task table, the second DDL task is split into a plurality of subtasks; According to the number of subtasks, a corresponding number of task execution units are invoked to execute the subtasks in parallel.

6. The method of claim 5, wherein, The invocation of the corresponding number of task execution units to execute the subtasks further includes: Creating a reorganization information table of the second DDL task; Recording meta information of the subtasks in the reorganization information table; The invocation of the corresponding number of task execution units to execute the subtasks includes: Invoking the task execution units to execute the corresponding subtasks according to the meta information of the corresponding subtasks in the reorganization information table.

7. The method of claim 6, wherein, The meta information of the subtasks includes at least one of data positions at which the subtask starts scanning and ends scanning, a node identifier of the target computing node, and an execution status of the subtask.

8. The method of claim 1, wherein, The task parallel processing rule is implemented based on a structured query language (SQL) statement. The determination of the second DDL task that meets the pre-determined task parallel processing rule from the job task table includes: Selecting a reference DDL task from the job task table, writing a task identifier and a schema identifier of the reference DDL task into the task parallel processing rule, and if it is determined that the task parallel processing rule meets a pre-set condition, taking the reference DDL task as the second DDL task.

9. The method according to any one of claims 1 to 8, characterized in that, The parallel processing of the second DDL task and the first DDL task further includes: After the processing of the second DDL task is completed, the related information of the second DDL task is moved from the job task table to a task history table, and the task history table includes fields in the job task table.

10. The method of claim 1, wherein, The parallel processing of the second DDL task and the first DDL task further includes: Configuring an execution strategy of the second DDL task, and the execution strategy includes information of a target task execution unit used to execute the first DDL task.

11. A target computing node in a database system, characterized by: It includes: A job task table module is configured to obtain a job task table, the job task table is used to record related information of an unprocessed DDL task, and the related information includes an order in which the DDL task enters a database system and a data schema in which a change object corresponding to the DDL task is located. A task determining module is configured to determine a first DDL task being executed, determine a second DDL task from the job task table that meets a pre-determined task parallel processing rule, the task parallel processing rule being related to an order in which DDL tasks enter a database system and a data mode in which a change object corresponding to a DDL task is located; and the task parallel processing rule is implemented based on a select type SQL statement. A parallel processing module is configured to parallel process the second DDL task and the first DDL task. The task parallel processing rule includes: Selecting a task that does not perform data reorganization and is submitted earliest from unprocessed DDL tasks as a DDL task to be executed in parallel.

12. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 11. The processor executes the computer program to implement the steps of the parallel processing method of any one of claims 1-10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the parallel processing method of any one of claims 1-10.

Images