Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence cloud computing, in particular to the technical fields of a distributed database technology, data access, data table change and the like. According to the specific implementation scheme, in response to received broadcast information for distributed nodes, table structure information for a first target table is obtained based on the broadcast information, the first target table is obtained by executing a table structure change task on a first original table, and the table structure information represents a table structure of the first target table; the table structure information is utilized to update the table structure of the second original table to obtain a second target table, and the first original table and the second original table are deployed on different distributed nodes and belong to the same data table; in response to determining a transaction processing result about the data access transaction, determining a data source accessed by the data access transaction; and if the data source includes the second original table, rollback the data access transaction.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and cloud computing technology, and in particular to the fields of distributed database technology, data access, and data table modification, specifically to data processing methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] Distributed databases, also known as distributed clusters, are database systems that store data across multiple distributed nodes. Distributed databases are characterized by data distribution, high availability, scalability, and consistency. When modifying the table structure of data tables stored in a distributed database, ensuring that these multiple characteristics are maintained simultaneously has become a key research focus. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a data processing method is provided, comprising: responding to receiving broadcast information for a distributed node; obtaining table structure information for a first target table based on the broadcast information, wherein the first target table is obtained by performing a table structure modification task on a first original table, and the table structure information characterizes the table structure of the first target table; updating the table structure of a second original table using the table structure information to obtain a second target table, wherein the first original table and the second original table are deployed on different distributed nodes and belong to the same data table; determining the data source accessed by the data access transaction in response to determining the transaction processing result of a data access transaction; and rolling back the data access transaction if the data source includes the second original table.

[0005] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a broadcast module, configured to, in response to receiving broadcast information for a distributed node, obtain table structure information for a first target table based on the broadcast information, wherein the first target table is obtained by performing a table structure modification task on a first original table, and the table structure information characterizes the table structure of the first target table; an update module, configured to update the table structure of a second original table using the table structure information to obtain a second target table, wherein the first original table and the second original table are deployed on different distributed nodes and belong to the same data table; a transaction processing module, configured to, in response to determining the transaction processing result of a data access transaction, determine the data source accessed by the data access transaction; and a rollback module, configured to roll back the data access transaction if the data source includes the second original table.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This illustration schematically shows an exemplary system architecture to which data processing methods and apparatus can be applied according to embodiments of the present disclosure;

[0012] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 3 The illustration shows a schematic diagram of a data table update interaction according to an embodiment of the present disclosure;

[0014] Figure 4 The illustration shows a schematic diagram of the metadata status update process according to an embodiment of the present disclosure;

[0015] Figure 5 A timing diagram illustrating the execution of a table structure change task according to an embodiment of the present disclosure is shown.

[0016] Figure 6 A block diagram of a data processing apparatus according to embodiments of the present disclosure is schematically shown; and

[0017] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] Data Definition Language (DDL) operations, also known as table structure change operations, can include adding or deleting table fields, adding indexes, and changing field types. With the continuous growth of business volume, table structure change tasks are becoming increasingly frequent. Traditional table structure change methods often employ table-level locking mechanisms, but these frequently lead to prolonged downtime of the distributed cluster, severely impacting service availability and potentially causing data consistency risks when DDL operations fail.

[0020] Distributed clusters typically consist of multiple distributed nodes, each of which needs to cache the same table structure information (also known as schema information) for performance reasons. When data tables change, due to complex distributed environment issues such as response timing, distributed node failures, or network isolation, it is difficult to guarantee that all distributed nodes synchronize to the same version of the table structure information at the same time. Therefore, traditional DDL operations can no longer meet the high availability and consistency requirements of distributed scenarios.

[0021] In view of this, the present disclosure provides a data processing method, comprising: responding to receiving broadcast information for a distributed node, obtaining table structure information for a first target table based on the broadcast information, wherein the first target table is obtained by performing a table structure modification task on a first original table, and the table structure information characterizes the table structure of the first target table; updating the table structure of a second original table using the table structure information to obtain a second target table, wherein the first original table and the second original table are deployed on different distributed nodes and belong to the same data table; responding to determining the transaction processing result of a data access transaction, determining the data source accessed by the data access transaction; and rolling back the data access transaction if the data source includes the second original table.

[0022] The data processing method provided by the embodiments of this disclosure can support online table structure changes based on the characteristics of distributed clusters, thereby enabling concurrent execution of table structure changes and data access transactions, and improving the high availability, high consistency and high scalability of data tables. It can achieve safe and efficient dynamic changes to table structures without interrupting data access transactions, effectively solving the problems of consistency and availability of changes in a distributed environment.

[0023] Figure 1 An exemplary system architecture for which data processing methods and apparatus can be applied according to embodiments of this disclosure is illustrated.

[0024] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the data processing methods and apparatus can be applied may include a terminal device, but the terminal device can implement the data processing methods and apparatus provided by the embodiments of this disclosure without interacting with the server.

[0025] like Figure 1 As shown, the system architecture according to this embodiment may include a terminal device 101, a network, and a distributed cluster 102. The network serves as a medium for providing a communication link between the terminal device 101 and the distributed cluster 102. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0026] Users can use terminal device 101 to interact with distributed cluster 102 via a network to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (for example only).

[0027] Terminal device 101 can be various electronic devices with a display screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0028] The distributed cluster 102 can be a distributed system or a cloud server cluster, and may include multiple distributed nodes 1022_1, ..., 1022_N-1. N is greater than or equal to 3. Each distributed node can be a server that provides various services, such as a backend management server that supports the content browsed by the user using terminal device 101 (for example only). The backend management server can analyze and process the received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0029] Optionally, in addition to including multiple distributed nodes 1022_1, ..., 1022_N-1, the distributed cluster 102 may also include a global metadata service manager 1021. The global metadata service manager 1021 is used to determine the master node 1022_N and record the table identification information, such as the version number, of the data tables deployed on each distributed node.

[0030] The master node 1022_N can be used to perform a table structure modification task on the first original table to obtain the first target table. In addition, it needs to send the storage address of the table structure information or the table structure information to the global metadata service manager 1021, so that the global metadata service manager 1021 can send broadcast information to multiple distributed nodes 1022_1, ..., 1022_N-1. This allows each distributed node to execute the data processing method provided in this embodiment based on the broadcast information to obtain the second target table.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and distributed clusters shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and distributed clusters can be included.

[0032] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0033] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0034] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0035] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0036] like Figure 2 As shown, the method includes operations S210~S240.

[0037] In operation S210, in response to receiving broadcast information for the distributed nodes, table structure information for the first target table is obtained based on the broadcast information.

[0038] In operation S220, the table structure of the second original table is updated using the table structure information to obtain the second target table.

[0039] In operation S230, in response to determining the transaction processing result regarding the data access transaction, the data source accessed by the data access transaction is determined.

[0040] In operation S240, if the data source includes a second original table, roll back the data access transaction.

[0041] The first target table is obtained by performing a table structure modification task on the first original table. The table structure information represents the table structure of the first target table. The first target table can be obtained by performing a table structure modification task on the first original table deployed on the master node online.

[0042] Table structure information, also known as schema information, is used to characterize the table structure of a data table. Specifically, it can include field names, field types, and relationships.

[0043] Table structure modification tasks, also known as Data Definition Language (DDL) operations, mainly include operations such as adding, deleting, changing field types, or adding indexes to a data table.

[0044] Optionally, executing table structure change tasks online can be understood as concurrently executing table structure change tasks and data access transactions on the first original table online.

[0045] Data access transactions, also known as Data Manipulation Language (DML) operations, mainly include operations such as inserting data, modifying data, and deleting data.

[0046] The broadcast message may include the storage address of the table structure information of the first target table and a prompt indicating that the table structure of the first original table has been updated. The broadcast message may be issued by the global metadata service manager to prompt multiple distributed nodes to update the table structure of the second original table deployed locally in a timely manner.

[0047] The first and second original tables are deployed on different distributed nodes but belong to the same data table. For example, the first and second original tables are different data shards of the same data table.

[0048] Distributed nodes can respond to receiving this broadcast information and obtain the table structure information of the first target table.

[0049] Because executing table structure change tasks takes a long time, a distributed node, such as the master node, can perform the task to obtain the first target table and its structure information. Other distributed nodes can then retrieve this structure information and update the structure of their respective second original tables stored locally. This ensures data consistency across all distributed nodes in the distributed cluster while reducing the execution time and complexity of table structure changes.

[0050] In an optional embodiment, when updating the second original table using table structure information, a table lock operation can be performed on the second original table to ensure data consistency. The table lock operation may include applying a table access lock to the second original table, for example, disallowing reading and writing to the second original table. After applying a table lock to the second original table, when a distributed node receives a data access transaction, it needs to pause processing to wait for the second original table to be updated to the second target table before resuming the data access transaction.

[0051] In another alternative embodiment, in response to receiving a data access transaction, the data access transaction can be executed directly to obtain the transaction processing result. The data source accessed by the data access transaction is determined. If the data source includes the second original table, the data access transaction is rolled back. If the data source does not include the second original table, the data access transaction is determined to have been completed.

[0052] Compared to the example of performing table lock operations and suspending data access transactions, the data processing method provided by the embodiments of this disclosure can determine whether to roll back the data source after the data access transaction processing is completed, thereby avoiding the rollback of a large number of data access transactions, improving the concurrent execution capability of table structure updates and data access transactions, and avoiding interruption or delay of data access transactions.

[0053] Figure 3 A schematic diagram illustrating a data table update interaction according to an embodiment of the present disclosure is shown.

[0054] like Figure 3 As shown, terminal device 101 sends a table structure change request to distributed cluster 102. In response to the request, a table structure change task is generated and added to the task queue of metadata node 1022_N+1 in distributed cluster 102. Master node 1022_N pulls and executes the table structure change task from the task queue, obtaining the first target table and its table structure information. Master node 1022_N stores the table structure information in a predetermined storage space and sends a notification to the global metadata service manager 1021 that the first original table has been updated and its structure information has been stored in the predetermined storage space. The global metadata service manager 1021 then generates and sends a broadcast message to prompt each distributed node 1022_1, ..., 1022_N-1 to obtain the table structure information and update the second original table.

[0055] Optionally, the distributed nodes obtaining the target table based on broadcast information may include: determining the storage address of the table structure information based on the broadcast information; and obtaining the table structure information through the storage address.

[0056] Optionally, the pre-defined storage space corresponding to the storage address can be deployed in the global metadata service manager, but it is not limited to this. It can also be deployed in any distributed node, as long as it can be shared among the distributed nodes.

[0057] Utilize Figure 3 The method shown allows for updating data tables across all distributed nodes in a distributed cluster for tasks involving changes to the table structure. In this distributed cluster, only the master node needs to execute the table structure task, while other distributed nodes obtain the table structure information and update the second original table based on that information. This improves the consistency of the table structure across the entire distributed cluster and simplifies the operation.

[0058] The following section will describe in detail the table structure change task performed by the master node.

[0059] According to embodiments of this disclosure, the first target table is generated as follows: In response to receiving a table structure change task, a temporary table is generated. Based on the temporary table, the table structure change task is executed. In response to the completion of the table structure change task, the first target table is obtained.

[0060] According to embodiments of this disclosure, the operation of generating a temporary table can be understood as preparatory work before executing a table structure change task. A first target table can be generated based on the temporary table. The table structure of the temporary table matches the table structure indicated by the table structure change task. This achieves the table structure change task.

[0061] According to one embodiment of this disclosure, a table structure change task may include a field type change task. Before executing the table structure change task, the data processing method may further include: updating the metadata state to an exclusive state. If the metadata state has been updated to an exclusive state, a temporary table is generated.

[0062] The exclusive state represents a state in which access to the first original table is not allowed and table structure changes to the first original table are not allowed.

[0063] Adding exclusive locks allows you to request MDL exclusive locks on each distributed node, preventing any further changes to the original table structure or data access transactions within the entire distributed cluster. This ensures data consistency across the distributed cluster.

[0064] During the execution of table structure change tasks, a "DML priority strategy" can be supported. Under this strategy, the distributed cluster adopts a distributed metadata lock (MDL) mechanism to ensure the priority execution of data access transactions and reduce the interruption of data access transactions caused by DDL operations.

[0065] Metadata locking mechanisms refer to adding metadata locks to the metadata of the first original table, thereby restricting changes to the table structure of the first original table while not affecting access to the first original table. Optionally, metadata locks can include exclusive locks, etc. Adding an exclusive lock updates the metadata status of the first original table to an exclusive state. An exclusive state can represent a state where changes to the table structure of the first original table are not allowed, and access to the first original table is not allowed.

[0066] Through the MDL locking mechanism, data access transactions can obtain high execution priority in high-concurrency scenarios, ensuring the continuity and stability of transactions.

[0067] Optionally, before updating the metadata state of the first original table targeted by the table structure change task, the data processing method may further include: determining whether the metadata locks of all distributed nodes in the distributed cluster are in a released state. If not, then waiting for the metadata locks of all distributed nodes to be released.

[0068] Optionally, before updating the metadata status of the first original table targeted by the table structure change task, the data processing method may further include: determining whether there are any anomalies in each distributed node of the distributed cluster. If an anomaly exists, wait for a predetermined extended period. If normal, directly update the metadata status of the first original table targeted by the table structure change task.

[0069] Figure 4 The illustration shows a schematic diagram of the metadata status update process according to an embodiment of the present disclosure.

[0070] like Figure 4 As shown, after confirming that the metadata locks of all distributed nodes CN1 and CN2 have been released, the master node CN_Owner requests exclusive locks from each distributed node CN1 and CN2 to update the metadata status to an exclusive state. A temporary table is generated. Metadata locks are then requested to be released from each distributed node CN1 and CN2. A table structure change task is executed. In response to the completion of the table structure change task, the first target table is obtained.

[0071] According to another embodiment of this disclosure, the table structure change task may include an index addition task. Before executing the table structure change task, the data processing method may further include: adding index field columns to the index table of the first original table, and generating a temporary table, provided that the metadata status has been updated to an exclusive state.

[0072] According to embodiments of this disclosure, the table structure of the temporary table matches the table structure indicated by the table structure change task. To ensure data consistency, data access transactions can be restricted during the generation of the temporary table structure, thereby improving data table consistency. Furthermore, since the table structure change task is a new index task, an indexed field column can be added to the index table of the first original table to create a temporary table, thereby improving processing efficiency and simplifying temporary table operations.

[0073] The previous section explained how to update the metadata status when performing a table structure change task. The following section will explain how to perform the table structure change task itself.

[0074] According to embodiments of this disclosure, performing a table structure change task may include: dividing the data in a first original table into blocks to obtain multiple data blocks. If the block status of a data block has been updated to a block lock state, the data block in the block lock state is copied to a temporary table.

[0075] The block lock state indicates a state where write operations on the data block are not allowed.

[0076] For existing data in the first original table, an asynchronous backfill mechanism can be used. For example, the existing data can be divided into multiple small batches of data blocks, locked in batches, and backfilled to a temporary table. Locking only a limited range of data each time reduces the probability of conflicts with concurrent online write operations and effectively prevents business blockage.

[0077] To prevent data consistency issues arising from data migration operations that copy data blocks to a temporary table in batches during backfilling, and from incremental operations, a block lock can be added to each backfilled data block. After adding a block lock, the data blocks in the migration state of the first original table are in a block-locked state. The block lock state indicates that write operations are not allowed on the data block. This ensures that backfilling and incremental write operations are mutually exclusive, achieving strong data consistency.

[0078] According to embodiments of this disclosure, during the execution of a table structure change task, the data processing method may further include: in response to receiving a data update task for a target data block, determining the block state of the target data block; and, if the block state characterization has been updated to the block lock release state, performing data update tasks on both the first original table and the temporary table.

[0079] The block lock release state indicates that a write operation is allowed on the target data block.

[0080] When performing a table structure change task, specifically, when performing a migration operation on the target data block, the data update task can be paused. The data to be processed indicated by the data update task is treated as incremental data, and processing is paused. The block status of the target data block is determined. When the block status indicates that the block lock is in a released state, allowing write operations on the target data block, both the original table and the temporary table execute data update tasks.

[0081] Optionally, a dual-write mechanism can be used, such as synchronizing every insert, delete, and update operation on the first original table to a temporary table in real time. This dual-write mechanism ensures strong consistency between the first original table and the first target table, guaranteeing real-time data synchronization and correct mapping regardless of the write operation pressure. It eliminates issues such as noise and omissions caused by concurrent incremental and existing data, meeting the high demands for data security and stability in industries such as finance and e-commerce.

[0082] According to the embodiments of this disclosure, compared with the traditional full table locking or one-time batch backfilling scheme, combining the asynchronous backfilling mechanism with the dual-write mechanism can shorten the duration of the table lock state of the first original table, effectively improve the throughput and availability of data update tasks during online DDL operations, and significantly optimize the table structure change experience in high-concurrency environments.

[0083] Figure 5 A timing diagram illustrating the execution of a table structure change task according to an embodiment of the present disclosure is shown.

[0084] like Figure 5 As shown, the master node CN_Owner divides the data in the first original table into multiple data blocks according to a predetermined data volume. Following the data block arrangement order of the first original table, the current backfill data block is determined from these multiple data blocks. The block status of the current backfill data block is updated to a block lock state, and the data block in the block lock state is copied to a temporary table. During the migration process, execution is paused in response to receiving a data update task for the current backfill data block in the block lock state. After the data block has been completely copied to the temporary table, the block status of the current backfill data block is updated to a block lock released state. Data update tasks are executed on both the first original table and the temporary table, completing a double write. After confirming that multiple data blocks have completed the migration, the temporary table is used as the first target table, and the first original table is deleted, releasing storage space.

[0085] Optionally, the data blocks can be divided equally according to the amount of data in the first original table, but it is not limited to this. They can also be divided according to a predetermined amount of data, or according to the predetermined migration duration of each batch, as long as the risk of concurrency conflict with the data update task is reduced.

[0086] The above section explained how to perform table structure change tasks. The following section will explain how to perform data access transactions.

[0087] According to embodiments of this disclosure, for example, Figure 2 The operation S220 shown, which determines the data source accessed by the data access transaction, may include: determining data source identification information to identify the data source based on the transaction processing result.

[0088] Data source identification information is added to the transaction result when processing data access transactions. This information can include the table identifier (e.g., the table name) and table structure information such as the version number. Essentially, any information that uniquely identifies the data source is acceptable.

[0089] To accommodate the concurrent execution of updates to the second original table obtained after online DDL operations and data access transactions, this disclosure proposes a new data processing method. During the execution of a data access transaction, the data source accessed by the data access transaction is determined, and the data source identification information is recorded and added to the transaction processing result.

[0090] This allows us to determine whether the data source includes the second original table, based on the table identifier information of the second original table and the data source identifier information, after determining the transaction processing result.

[0091] This ensures that table updates and transaction processing are executed concurrently, while data access transactions to unrelated second original tables remain unaffected. This achieves table-level version constraints, improves concurrency capabilities, and reduces rollbacks of unrelated transactions.

[0092] According to embodiments of this disclosure, after determining whether the data source includes the second original table, the data processing method may further include: if the data source does not include the second original table, sending the transaction processing result to the recipient indicated by the data access transaction.

[0093] This feature detects whether the data source on which a data access transaction depends has changed. If it has changed, the data access transaction is rolled back; otherwise, it is allowed to commit normally. Rollback is only performed on data access transactions related to the second original table involved in DDL operations; data access transactions on unrelated tables are unaffected, thus effectively reducing the probability of unrelated data access transactions being rolled back. This allows the receiver to be unaware of online DDL operations, avoiding a large number of transaction blocks, and is suitable for high write concurrency scenarios.

[0094] The data processing method provided in this disclosure can support table structure change tasks by updating metadata status in complex scenarios such as index addition, data table field changes, and table routing adjustments. This enables online table structure changes, improves the concurrent execution of data access transactions and table structure change tasks, and enhances the stability of high-concurrency application scenarios. Furthermore, the combination of a dual-write mechanism and an asynchronous backfill mechanism shortens table locking time and improves data consistency. A lightweight implementation method that does not rely on third-party components, such as the data migration tool DTS (Data Transformation Service), is employed. Additionally, utilizing distributed nodes to replicate table structure information via broadcast simplifies the synchronization of table structure change tasks across multiple distributed nodes in a distributed cluster, improving processing efficiency. Moreover, by adding data source identification information to data access transactions, concurrent updates to the second original table are supported, avoiding rollbacks of data access transactions unrelated to the second original table, reducing latency, and improving the user experience for business recipients.

[0095] The data processing method provided in this disclosure is applicable to distributed clusters in industries such as finance, e-commerce, telecommunications, and the internet, where data consistency, availability, and scalability are extremely important. Furthermore, it can be widely used to support cloud database service platforms, ensuring high concurrency capabilities in cloud-native environments. Additionally, it is suitable for use in big data infrastructure such as data platforms and data warehouses, ensuring online availability for complex operations such as structure changes and index optimization.

[0096] Figure 6 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0097] like Figure 6 As shown, the data processing device 600 includes: a broadcast module 610, an update module 620, a transaction processing module 630, and a rollback module 640.

[0098] The broadcast module 610 is used to respond to receiving broadcast information for distributed nodes, and to obtain table structure information for a first target table based on the broadcast information. The first target table is obtained by performing a table structure modification task on a first original table, and the table structure information represents the table structure of the first target table.

[0099] The update module 620 is used to update the table structure of the second original table using the table structure information to obtain the second target table, wherein the first original table and the second original table are deployed on different distributed nodes but belong to the same data table.

[0100] The transaction processing module 630 is used to determine the data source accessed by the data access transaction in response to determining the transaction processing result of the data access transaction.

[0101] Rollback module 640 is used to roll back data access transactions when the data source includes a second original table.

[0102] According to embodiments of this disclosure, the data processing apparatus further includes a first table generation module to generate a first target table.

[0103] According to embodiments of this disclosure, the first table generation module further includes: a temporary table generation submodule, a table modification submodule, and a first table generation submodule.

[0104] The temporary table generation submodule is used to generate a temporary table in response to a received table structure change task. The table structure of the temporary table matches the table structure indicated by the table structure change task.

[0105] The table change submodule is used to perform table structure change tasks based on temporary tables.

[0106] The first table generation submodule is used to obtain the first target table in response to the completion of the table structure change task.

[0107] According to embodiments of this disclosure, the table change submodule includes: a block unit and a copy unit.

[0108] A block unit is used to divide the data in the original table into blocks, resulting in multiple data blocks.

[0109] The copying unit is used to copy data blocks in the block lock state to a temporary table when the block state of the data block has been updated to the block lock state. The block lock state indicates that write operations on the data block are not allowed.

[0110] According to embodiments of this disclosure, the temporary table generation submodule includes: a first temporary table generation unit.

[0111] The first temporary table generation unit is used to perform data update tasks on both the first original table and the temporary table when the metadata status has been updated to the exclusive state. The block lock release status represents the state that allows write operations to be performed on the target data block.

[0112] According to embodiments of this disclosure, the table structure change task includes a field type change task. The data processing apparatus further includes a first temporary table generation module.

[0113] The first temporary table generation module is used to generate a temporary table when the metadata status has been updated to an exclusive status. The exclusive status represents a state in which access to the first original table is not allowed and table structure changes to the first original table are not allowed.

[0114] According to embodiments of this disclosure, the table structure change task includes a new index task. The temporary table generation submodule includes a second temporary table generation unit.

[0115] The second temporary table generation unit is used to add index field columns to the index table of the first original table and generate a temporary table when the metadata status has been updated to the exclusive status. The exclusive status represents a state in which access to the first original table is not allowed and table structure changes to the first original table are not allowed.

[0116] According to embodiments of this disclosure, the table update module includes an address determination submodule and a target table determination submodule.

[0117] The address determination submodule is used to determine the storage address of table structure information based on broadcast information.

[0118] The target table determination submodule is used to obtain table structure information through storage address.

[0119] According to embodiments of this disclosure, the transaction processing module includes a data source determination submodule.

[0120] The data source determination submodule is used to determine the data source identification information to identify the data source based on the transaction processing results.

[0121] Data source identification information is added to the transaction processing result when processing data access transactions.

[0122] According to embodiments of this disclosure, the data processing apparatus further includes a data source association determination module.

[0123] The data source association determination module is used to determine whether the data source includes the second original table based on the table identifier information of the second original table and the data source identifier information.

[0124] According to embodiments of this disclosure, the data processing apparatus further includes a submission module.

[0125] The commit module is used to send the transaction results to the recipient indicated by the data access transaction if the data source does not include the second original table.

[0126] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0128] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0129] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0130] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0132] Multiple components in device 700 are connected to input / output (I / O) interface 705, including: input unit 706, such as a keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: In response to receiving a broadcast message for a distributed node, based on the broadcast message, table structure information for a first target table is obtained, wherein the first target table is obtained by performing a table structure modification task on a first original table, and the table structure information characterizes the table structure of the first target table; The table structure of the second original table is updated using the table structure information to obtain the second target table, wherein the first original table and the second original table are deployed on different distributed nodes but belong to the same data table; In response to determining the transaction processing result regarding the data access transaction, the data source accessed by the data access transaction is determined; and If the data source includes the second original table, roll back the data access transaction.

2. The method according to claim 1, wherein, The first target table is generated in the following way: In response to receiving the table structure change task, a temporary table is generated, the table structure of which matches the table structure indicated by the table structure change task; Based on the temporary table, execute the table structure change task; as well as In response to the completion of the table structure change task, the first target table is obtained.

3. The method according to claim 2, wherein, The execution of the table structure change task includes: The data in the first original table is divided into blocks to obtain multiple data blocks; and If the block status of a data block has been updated to the block lock state, the data block in the block lock state is copied to the temporary table, wherein the block lock state indicates that write operations are not allowed on the data block.

4. The method according to claim 3, further comprising: In response to receiving a data update task for a target data block, determine the block status of the target data block; as well as When the block state representation has been updated to the block lock released state, the data update task is performed on both the first original table and the temporary table, wherein the block lock released state represents a state that allows write operations to be performed on the target data block.

5. The method according to any one of claims 2 to 4, wherein, The table structure change task includes field type change task; The generation of the temporary table includes: The temporary table is generated when the metadata status has been updated to an exclusive state, wherein the exclusive state represents a state in which access to the first original table is not allowed and table structure changes to the first original table are not allowed.

6. The method according to any one of claims 2 to 4, wherein, The table structure change task includes the task of adding a new index; The generation of the temporary table includes: When the metadata status has been updated to an exclusive status, an index field column is added to the index table of the first original table to generate the temporary table. The exclusive status represents a state in which access to the first original table is not allowed and table structure changes to the first original table are not allowed.

7. The method according to any one of claims 1 to 6, wherein, The step of obtaining table structure information for the first target table based on the broadcast information includes: Based on the broadcast information, determine the storage address of the table structure information; and The table structure information is obtained through the storage address.

8. The method according to any one of claims 1 to 6, wherein, Determining the data source accessed by the data access transaction includes: Based on the transaction processing result, determine the data source identification information used to identify the data source; The data source identification information is added to the transaction processing result when processing the data access transaction.

9. The method according to claim 8, further comprising: Based on the table identifier information of the second original table and the data source identifier information, determine whether the data source includes the second original table.

10. The method according to claim 1 or 9, further comprising: If the data source does not include the second original table, the transaction result is sent to the recipient indicated by the data access transaction.

11. A data processing apparatus, comprising: A broadcast module is used to respond to receiving broadcast information for distributed nodes, and based on the broadcast information, to obtain table structure information for a first target table, wherein the first target table is obtained by performing a table structure modification task on a first original table, and the table structure information represents the table structure of the first target table; The update module is used to update the table structure of the second original table using the table structure information to obtain the second target table, wherein the first original table and the second original table are deployed on different distributed nodes and belong to the same data table; The transaction processing module is configured to determine the data source accessed by the data access transaction in response to determining the transaction processing result regarding the data access transaction; and A rollback module is used to roll back the data access transaction if the data source includes the second original table.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images