Data processing method, electronic device, storage medium, and program product

Abstract

The specification provides a data processing method, an electronic device, a storage medium and a program product. The method is applied to a management node in a distributed storage system; the distributed storage system further includes a distributed database, a pre-write log and a CDC; the distributed database includes the management node and a storage node; the method includes: receiving a database operation request initiated by a client; wherein the database operation request includes a schema change operation for the distributed database; distributing the database operation request to a target storage node, and storing a correspondence relationship between an updated schema version identifier and a changed schema after the target storage node returns an execution result of the schema change operation; in response to a target schema version identifier sent by the CDC, querying a schema corresponding to the target schema version identifier from the stored correspondence relationship between each schema version identifier and a corresponding schema, and returning the queried schema to the CDC.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of Internet technology, and more particularly to a data processing method, electronic device, storage medium, and program product. Background Technology

[0002] In a distributed database, the database is actually partitioned into several independent database instances. Each database instance can generate its own changed data, which can be obtained by the CDC (Change Data Capture) module.

[0003] However, in a distributed environment, a logical database can actually consist of several physically separate and independent database instances, which are interconnected only through a network. Therefore, performing DDL (Data Definition Language) on a logical database translates into schema modifications for each database instance, which are then broadcast over the network to every database instance in the distributed database cluster.

[0004] If events across the entire distributed database cluster are placed on a single timeline, inconsistencies in the schema change times of storage nodes within the distributed database cluster can arise.

[0005] Furthermore, since the CDC module is an asynchronous module, it is not embedded in the online write chain. Therefore, the CDC module will always process the changed data generated by the write at some point after the actual write has occurred and been completed.

[0006] Therefore, in asynchronous consumption of CDC, there will be a problem where the schema at the actual time of occurrence is inconsistent with the schema at the actual time of consumption. Summary of the Invention

[0007] In view of the above, this specification provides one or more embodiments of a data processing method, an electronic device, a computer-readable storage medium, and a computer program product, to provide a method for ensuring data processing.

[0008] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0009] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a distributed storage system, the distributed storage system including a distributed database, a write-ahead log, and a CDC (Distributed Data Change Controller); the distributed database includes at least one management node and at least one storage node; wherein, the management node stores schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the method includes:

[0010] The management node receives database operation requests initiated by the client and distributes the database operation requests to the target storage node; wherein, the database operation requests include schema change operations or data write operations for the distributed database;

[0011] In response to the schema change operation of the database operation request, the target storage node updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation; further, it writes a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier;

[0012] The CDC reads the write-ahead log, asynchronously consumes executed database operation requests, and when it reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log, and processes the data write operation corresponding to the log between the schema change log and the next schema change log based on the queried schema.

[0013] Optionally, before writing the schema change log to the write-ahead log, the target storage node further includes:

[0014] The target storage node reports the updated schema version identifier and the changed schema to the management node;

[0015] The management node stores the correspondence between the updated schema version identifier and the changed schema.

[0016] Optionally, querying the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log includes:

[0017] The CDC sends the target schema version identifier recorded in the schema change log to the management node;

[0018] The management node queries the schema corresponding to the target schema version identifier from the stored mapping between each schema version identifier and its corresponding schema, and returns the queried schema to the CDC.

[0019] Optionally, the modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0020] According to a second aspect of the embodiments of this specification, a data processing method is provided, applied to a management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Catch-Up) mechanism; the distributed database includes at least one of the management nodes and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the method includes:

[0021] Receive database operation requests initiated by clients; wherein, the database operation requests include schema change operations for distributed databases;

[0022] The database operation request is distributed to the target storage node, and the correspondence between the updated schema version identifier returned by the target storage node after the schema change operation is performed and the changed schema is stored.

[0023] In response to the target schema sent by the CDC, the schema corresponding to the target schema version identifier is queried from the stored mapping between each schema version identifier and the corresponding schema, and the queried schema is returned to the CDC.

[0024] Optionally, the modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0025] According to a third aspect of the embodiments of this specification, a data processing method is provided, applied to a target storage node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Catch-Up). The distributed database includes at least one management node and at least one storage node; wherein the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the method includes:

[0026] Receive database operation requests distributed by the management node; wherein, the database operation requests include schema change operations for the distributed database;

[0027] In response to the schema change operation of the database operation request, based on the schema change data indicated by the schema change operation, the schema of the distributed database is updated and an updated schema version identifier is generated;

[0028] Write a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier.

[0029] Optionally, the modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0030] According to a fourth aspect of the embodiments of this specification, a data processing method is provided, applied to a CDC in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and the CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the method includes:

[0031] Read the write-ahead log and asynchronously consume executed database operation requests;

[0032] When the schema change log is read, the management node is queried for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log.

[0033] Based on the retrieved schema, perform data writing operations on the logs corresponding to the schema change logs between the schema change logs and the next schema change log.

[0034] According to a fifth aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Catch-Up). The distributed database includes at least one of the management nodes and at least one storage node; wherein the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the apparatus includes:

[0035] The receiving unit receives database operation requests initiated by the client; wherein, the database operation requests include schema change operations for the distributed database.

[0036] The storage unit distributes the database operation request to the target storage node and stores the correspondence between the updated schema version identifier returned by the target storage node after the schema change operation is performed and the changed schema.

[0037] The query unit, in response to the target schema version identifier sent by the CDC, queries the schema corresponding to the target schema version identifier from the stored mapping relationship between each schema version identifier and its corresponding schema, and returns the queried schema to the CDC.

[0038] According to a sixth aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a target management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Censor); the distributed database includes at least one management node and at least one storage node; wherein the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the apparatus includes:

[0039] The receiving unit receives database operation requests distributed by the management node; wherein, the database operation requests include schema change operations for the distributed database.

[0040] The processing unit, in response to the schema change operation of the database operation request, updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation.

[0041] The writing unit writes a schema change log into the write-ahead log; wherein the schema change log includes the updated schema version identifier.

[0042] According to a seventh aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a CDC in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and the CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the apparatus includes:

[0043] The read unit reads the write-ahead log and asynchronously consumes executed database operation requests;

[0044] When the query unit reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log.

[0045] The consumption unit processes the data writing operations corresponding to the logs between the schema change log and the next schema change log based on the queried schema.

[0046] According to an eighth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor implements the data processing method described in any embodiment of this specification by running the executable instructions.

[0050] According to a ninth aspect of an embodiment of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the data processing method described in any embodiment of this specification.

[0051] According to a tenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method described in any embodiment of this specification.

[0052] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0053] By designing an updated schema version identifier generated after a schema change operation, and inserting a schema change log entry recording this schema version identifier into the write-ahead log, the CDC can asynchronously consume database operation requests by reading logs from the write-ahead log. It can then distinguish between logs with different schemas based on the read schema change logs, and further obtain the correct schema correspondence between the current schema change log and the next schema change log by querying the management node.

[0054] On the one hand, schema version identifiers are used to identify the correspondence between database operation requests at different time periods and the corresponding schemas, thereby ensuring schema consistency during asynchronous CDC consumption.

[0055] On the other hand, it does not rely on a centralized TSO component to sort global database operation requests, thus avoiding performance limitations of the distributed storage system caused by the single point of failure of the centralized TSO component.

[0056] On the other hand, inserting schema change logs into the write-ahead log and storing version identifiers of each schema change log on the management node requires fewer resources, enabling lightweight and efficient data processing.

[0057] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0058] Figure 1 This is a schematic diagram illustrating DDL execution in a distributed database, provided by an exemplary embodiment.

[0059] Figure 2 This is a schematic flowchart of a data processing method applied to a distributed storage system, provided in an exemplary embodiment.

[0060] Figure 3 This is a schematic diagram illustrating a schema version identifier update provided in an exemplary embodiment;

[0061] Figure 4 This is a schematic diagram of a schema version identifier list provided in an exemplary embodiment;

[0062] Figure 5 This is a schematic diagram of a database operation request provided in an exemplary embodiment;

[0063] Figure 6 This is a schematic diagram of a distributed database process provided in an exemplary embodiment;

[0064] Figure 7 This is a flowchart illustrating a data processing method with a management node as the execution subject, provided in an exemplary embodiment.

[0065] Figure 8 This is a schematic flowchart of a data processing method with a storage node as the execution subject, provided in an exemplary embodiment.

[0066] Figure 9 This is a schematic flowchart of a data processing method with CDC as the execution subject, provided in an exemplary embodiment.

[0067] Figure 10 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment;

[0068] Figure 11 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment;

[0069] Figure 12 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment;

[0070] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0072] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0073] To facilitate understanding of the embodiments in this specification, some technical concepts involved in the embodiments are introduced below.

[0074] CDC: Change Data Capture. CDC is a database technology used to capture any insert, update, or delete operations that occur in a database table and record historical information about these changes.

[0075] A schema refers to the organizational structure of data in a database. It defines the logical view of objects in the database, including the structure and attributes of elements such as tables, views, indexes, stored procedures, and triggers. In the specification, it mainly refers to the structure of data tables, that is, the structured collection used to store data. Each table has its own columns, which define the data type and constraints.

[0076] DDL: Data Definition Language. This is a language used to define the structure of a database, including creating, modifying, and deleting database objects such as tables, indexes, and views.

[0077] TSO: Timestamp Ordering. This is a mechanism used to ensure the time order and consistency of transactions in a distributed system.

[0078] WAL: Write-Ahead Logging. This is a database technique used to ensure that data can be correctly restored during fault recovery. It first records changes to a log before applying them to the database.

[0079] In a distributed storage system that includes distributed databases (such as TiDB, PolarDB-X, etc.), the distributed database can actually be partitioned into several independent database instances, which can be called nodes. Based on the different functions of the nodes, the distributed database can be divided into management nodes and storage nodes.

[0080] There can be one or more management nodes. Management nodes are generally used to distribute database operation requests (also known as DDL requests) initiated by clients to storage nodes. For example, based on a load balancing strategy, requests can be distributed to storage nodes that are less busy.

[0081] Storage nodes are the nodes that actually operate on the distributed database. For example, storage nodes can write data to the distributed database or change its structure (such as modifying fields in a data table). This latter type of operation, which changes the structure of the distributed database, is usually called a schema change operation.

[0082] Regardless of the type of database operation request, it will be received by the CDC or CDC module (hereinafter collectively referred to as CDC) in the distributed storage system. CDC is generally used for data analysis, data mining, data aggregation and other related business after data is written. Therefore, CDC is not embedded in the online write chain, but is an asynchronous module. That is, CDC always processes the changed data corresponding to the executed database operation request after the storage node has actually completed the database operation request.

[0083] like Figure 1 As shown, after a DDL request is sent to the management node, the management node can distribute the DDL request to different storage nodes. After each storage node has completed its own DDL request, the management node summarizes the requests and sends feedback to the client, indicating that the DDL request has been completed.

[0084] The client initiates a DDL request to the distributed database at time t0. Each storage node executes its own DDL request at times t1, t2, and t3, respectively. Finally, at time tn, the management node receives the execution results from all storage nodes and returns a summary execution result to the client. From the perspective of the external client or management node, only time t0 or t1 is perceived; the actual execution times of the DDL requests by each storage node are not known as t1, t2, and t3.

[0085] Since CDC is an asynchronous module, a problem arises: the schema of a DDL request when it actually occurs may be inconsistent with the schema of CDC when it is asynchronously consumed.

[0086] To give a simple example:

[0087] At time t0, the data type of column A represented by the schema is int. In response to a DDL request A that writes a value of 1 to column A, a row of data with an int value of 1 is written to column A; and a corresponding log entry with an int value of 1 will be generated in the write-ahead log.

[0088] At time tn, in response to another DDL request B that changes the data type of column A to string, the data type of column A is changed from int to string.

[0089] At time t(n+1), the CDC consumes DDL request A. At this time, the data type of column A in the log read from the write-ahead log is still int (that is, the data type of column A represented by the schema is int). However, the data type of column A represented by the actual distributed database schema is already string. Because the two schemas are inconsistent, the CDC will be unable to consume DDL request A correctly.

[0090] To address the schema inconsistency issue that exists when CDC consumes DDL requests in a distributed database scenario, an additional TSO component is introduced in the relevant technology. Each DDL request needs to obtain a globally unique timestamp from the TSO component, and the entire distributed storage system defines the time order of each DDL request based on the timestamp of the TSO component.

[0091] However, the TOS component is centralized, resulting in a single point of performance bottleneck. This bottleneck, like the "barrel principle," limits the performance and processing efficiency of the distributed storage system. Furthermore, the TOS component itself is a relatively complex system, with high costs associated with its integration and use.

[0092] It should be noted that this invention targets distributed storage systems and distributed databases. In a single-machine database, since only one server processes user requests, DDL requests are ordered within the single machine. Therefore, even if CDC asynchronously consumes DDL requests, it will not face the problem of schema inconsistency.

[0093] In distributed scenarios, CDC asynchronous consumption of DDL requests faces the problem of schema inconsistency. This document provides a solution that can still guarantee the schema consistency of CDC without relying on the TSO component. This solution is lightweight and has a fast response speed.

[0094] Please refer to the following. Figure 2 The diagram shows a flow chart of a data processing method applied to a distributed storage system. Figure 2 As shown, the distributed storage system may include a distributed database, a write-ahead log (WAP) system, and a CDC (Distributed Data Acquisition System); the distributed database includes at least one management node and at least one storage node. Specifically, the method may include the following steps:

[0095] Step 101: The client sends a database operation request to the management node in the distributed database.

[0096] Step 110: The management node receives a database operation request initiated by the client.

[0097] For example, the database operation request can be a schema change operation for a distributed database, or a data write operation to write data to a distributed database.

[0098] Step 120: The management node distributes the database operation request to the target storage node.

[0099] For example, the management node can distribute database operation requests to target storage nodes based on a preset distribution strategy. For instance, based on a load balancing strategy, database operation requests can be distributed to the target storage node with the lowest actual load, according to the actual load of each storage node. Another example is distributing database operation requests to a specified target storage node based on the DDL request, and so on.

[0100] Step 130: In response to the schema change operation of the database operation request, the target storage node updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation.

[0101] In this embodiment, when a storage node receives a database operation request, if the database operation request is a data write operation, then the target storage node responds to the data write operation of the database operation request by writing the data to be written indicated by the data write operation to the distributed database, and generates a log corresponding to the data write operation in the write-ahead log accordingly.

[0102] If the database operation request is a schema change operation, in related technologies, the schema of the distributed database can be updated according to the schema change data indicated by the schema change operation, and the updated schema can be used to perform subsequent data write operations.

[0103] For example:

[0104] At time t0, the data type of column A represented by schema is int. In response to a DDL request A that writes a value of 1 to column A, a row of data with the value of 1 of type int is written to column A.

[0105] At time tn, in response to another DDL request B that changes the data type of column A to string, the data type of column A is changed from int to string, meaning that the data type of column A represented by the schema thereafter is string;

[0106] After time tn, all data write operations related to column A will write string data into column A according to the data type string represented by the schema at that time. This continues until a new schema change operation modifies the data type of column A.

[0107] Unlike related technologies, this specification, in addition to updating the schema of the distributed database according to the schema change data indicated by the schema change operation and using the updated schema to perform subsequent data write operations, also generates an updated schema version identifier and step 140.

[0108] This manual adds a unique schema version identifier to the schema after each schema change operation in a distributed database. For example... Figure 3 As shown, the modified schema version identifier may include a monotonically increasing schema version identifier based on the original schema version identifier.

[0109] In addition, the target storage node can also report the updated schema version identifier and the changed schema to the management node;

[0110] This allows the management node to store the correspondence between the updated schema version identifier and the changed schema.

[0111] In this way, it can be ensured that the management node can maintain the various schema version identifiers and corresponding schemas generated in the distributed database.

[0112] Step 140: The target storage node writes a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier.

[0113] For example, the write-ahead log may include the aforementioned write-album log (WAL), and the following embodiments will use the write-album log as an example for illustration.

[0114] It should be noted that after a storage node performs a data write operation, WAL will automatically generate relevant logs, but it will not automatically generate logs corresponding to schema change operations.

[0115] Therefore, in this specification, the storage node can actively write a special log, namely the schema change log, to the WAL before or after a schema change, using a hook function (such as the hook function), and use this schema change log to mark the latest schema version.

[0116] This specification also stores and maintains the schema version identifiers of the distributed database in the management node. Each schema version identifier corresponds to a schema after a schema change event is executed.

[0117] like Figure 4 As shown, a list (SchemaVersionMap) storing schema version identifiers is constructed in the management node. This list records the identifiers of each schema version generated historically (i.e., Figure 4 The version number in the schema, and the schema details corresponding to each schema version identifier.

[0118] Due to the inherent ordering of the Write-Ahead Log (WAL), data write logs prior to being written to the schema change log will always use the old schema encoding. All data write logs between the current schema change log and the next schema change log will also necessarily use the schema encoding of that schema change log. Therefore, based on the schema version identifier, the corresponding schema can always be accurately located in the management node, allowing the processing of DDL requests corresponding to the data write logs between the current and next schema version identifiers based on the found schema.

[0119] Step 150: CDC reads the write-ahead log and asynchronously consumes the executed database operation requests.

[0120] As mentioned earlier, CDC is an asynchronous module that can asynchronously consume DDL requests that have been executed by the management node by reading WAL.

[0121] Step 160: When the CDC reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log.

[0122] The WAL itself is ordered, meaning the logs are arranged in the order they were generated. Therefore, CDC can read the logs in the WAL from front to back according to the time sequence.

[0123] It should be noted that CDC can selectively read WAL based on its needs. For example, if the business needs to summarize DDL requests within the time period t100 to t1200, then CDC can read logs generated within this time period from WAL. Alternatively, CDC can also read logs within a specified log identifier range from WAL, such as logs from 100 to 200.

[0124] The following is combined Figure 5 The database operation requests shown are Figure 6 The diagram illustrating a distributed database process is further illustrated with examples:

[0125] Suppose that at time t0, the data type of column A represented by the schema is int, and at this time, a DDL request A is made to write the value 1 to column A, i.e. Figure 5 The function `Inset(1, true, "1")` will write a row of integers with the value 1 into column A; and as shown below... Figure 6 The following example shows the creation of a log entry in WAL: log 100:op insert,(1,true,“1”).

[0126] At time tn, in response to another DDL request B that changes the data type of column A to string, the data type of column A is changed from int to string, meaning that the data type of column A represented by the schema thereafter is string; and a log entry carrying the latest schema version identifier is inserted into the WAL via a hook, such as... Figure 6 The log 200:op schemachange,ver N+1 shown indicates that the latest schema version is identified as N+1.

[0127] At time tn+1, in response to a DDL request C that writes a value of 2 to column A, i.e. Figure 5 The function `Inset("2", true, "2")` will write a row of string data with the value 2 into column A; and as shown below... Figure 6 The example shown generates a log entry in WAL: log 201:op insert,(“2”, true, “2”).

[0128] During the process of CDC reading AWL, when reading logs from log100 to log199, the data type of column A is int as represented by the previous schema.

[0129] When the CDC reads log200, since this log is a schema change log, it can query the management node for the schema corresponding to N+1 based on the target schema version identifier (i.e., N+1) recorded in the schema change log. Figure 4 For example, the schema corresponding to N+1 is schema(N+1); and the data type of column A in schema(N+1) is string, while the others remain unchanged.

[0130] In an exemplary embodiment, as shown in step 130 above, after the target storage node updates the schema of the distributed database and generates an updated schema version identifier, it can report the updated schema version identifier and the changed schema to the management node; subsequently, the management node stores the correspondence between the updated schema version identifier and the changed schema. This ensures that the management node can maintain each schema version identifier generated in the distributed database and its corresponding schema.

[0131] Based on this, step 160, which queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log, may further include:

[0132] The CDC sends the target schema version identifier recorded in the schema change log to the management node;

[0133] The management node queries the schema corresponding to the target schema version identifier from the stored mapping between schema version identifiers and their corresponding schemas, and returns the retrieved schema to the CDC.

[0134] In this example, since the management node maintains the most complete mapping between each schema version identifier and its corresponding schema, the CDC only needs to send the target schema version identifier to the management node, and it can retrieve the schema corresponding to the target schema version identifier from the mapping maintained by the management node. The retrieved schema is the distributed database schema during the historical execution process of the database operation request that the CDC asynchronously consumes.

[0135] Step 170: The CDC processes the data writing operation corresponding to the log between the schema change log and the next schema change log based on the queried schema.

[0136] The steps mentioned above will still be used. Figure 5 and Figure 6 For example, after the CDC query finds schema(N+1), the data write operations corresponding to the data write logs between log201 and log299, which are between log200 and the next schema change log (let's say log300), can all be handled using schema(N+1).

[0137] Even if the distributed database has changed the data type of column A to bool in the next schema change log (log300), since the data type of column A represented by the queried schema (N+1) is the same as the data type of column A recorded in logs log201 to log299, which is string, there will be no schema inconsistency problem (which is equivalent to decoupling the comparison between the log schema and the real-time distributed database schema, and turning it into a comparison between the log schema and the schema corresponding to the schema version queried by the management node).

[0138] In summary, by designing an updated schema version identifier generated after a schema change operation, and inserting a schema change log entry recording this schema version identifier into the write-ahead log, the CDC can asynchronously consume database operation requests by reading logs from the write-ahead log. This allows it to distinguish between logs with different schemas based on the read schema change log, and further obtain the correct schema correspondence between the current schema change log and the next schema change log by querying the management node.

[0139] On the one hand, schema version identifiers are used to identify the correspondence between database operation requests at different time periods and the corresponding schemas, thereby ensuring schema consistency during asynchronous CDC consumption.

[0140] On the other hand, it does not rely on a centralized TSO component to sort global database operation requests, thus avoiding performance limitations of the distributed storage system caused by the single point of failure of the centralized TSO component.

[0141] On the other hand, inserting schema change logs into the write-ahead log and storing version identifiers of each schema change log on the management node requires fewer resources, enabling lightweight and efficient data processing.

[0142] After introducing the data processing method implementation examples of a multi-party interactive distributed storage system, the following sections will introduce the relevant data processing methods with management nodes, storage nodes, and CDC as the execution entities.

[0143] Please refer to Figure 7The diagram illustrates a data processing method with a management node as the execution entity; the method is applied to a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Censor); the distributed database includes at least one management node and at least one storage node; wherein, the management node stores the schema version identifiers of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the method includes:

[0144] Step 210: Receive a database operation request initiated by the client; wherein the database operation request includes a schema change operation for the distributed database.

[0145] Step 220: Distribute the database operation request to the target storage node, and store the correspondence between the updated schema version identifier returned by the target storage node after executing the schema change operation and the changed schema;

[0146] Step 230: In response to the target schema version identifier sent by the CDC, query the schema corresponding to the target schema version identifier from the stored mapping relationship between each schema version identifier and its corresponding schema, and return the queried schema to the CDC.

[0147] The modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0148] The steps in this embodiment can be referred to the foregoing. Figure 2 The relevant implementation examples will not be described in detail here.

[0149] Please refer to Figure 8 The diagram illustrates a data processing method with storage nodes as the execution entity; the target storage node is applied in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Catch-Up) mechanism; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores the schema version identifiers of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the method includes:

[0150] Step 310: Receive a database operation request distributed by the management node; wherein the database operation request includes a schema change operation for the distributed database.

[0151] Step 320: In response to the schema change operation of the database operation request, update the schema of the distributed database and generate an updated schema version identifier based on the schema change data indicated by the schema change operation;

[0152] Step 330: Write a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier.

[0153] In one exemplary embodiment, the database operation request further includes a data write operation, and the method further includes:

[0154] In response to the database operation request, a data write operation is performed to write the data to be written as indicated by the data write operation to the distributed database, and a log corresponding to the data write operation is generated in the write-ahead log accordingly.

[0155] In one exemplary embodiment, the modified schema version identifier includes a schema version identifier that is monotonically incremented based on the original schema version identifier.

[0156] The steps in this embodiment can be referred to the foregoing. Figure 2 The relevant implementation examples will not be described in detail here.

[0157] Please refer to Figure 9 The diagram illustrates a data processing method using CDC as the execution entity; CDC is applied to a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores the schema version identifiers of each schema version identifier of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the method includes:

[0158] Step 410: Read the write-ahead log and asynchronously consume the executed database operation requests;

[0159] Step 420: When the schema change log is read, based on the target schema version identifier recorded in the schema change log, query the management node for the schema corresponding to the target schema version identifier;

[0160] Step 430: Based on the queried schema, process the data writing operation corresponding to the log between the schema change log and the next schema change log.

[0161] The steps in this embodiment can be referred to the foregoing. Figure 2 The relevant implementation examples will not be described in detail here.

[0162] After introducing the method embodiments, the device embodiments corresponding to the method embodiments are further introduced below.

[0163] Figure 10 An example is provided in the structural block diagram of a data processing apparatus corresponding to the aforementioned Figure 7 The illustrated data processing method embodiment. Specifically, this data processing device can be applied to a management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC; the distributed database includes at least one of the management nodes and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; and the device may include a receiving unit 810, a storage unit 820, and a query unit 830.

[0164] The receiving unit 810 receives a database operation request initiated by the client; wherein the database operation request includes a schema change operation for a distributed database.

[0165] Storage unit 820 distributes the database operation request to the target storage node and stores the correspondence between the updated schema version identifier returned by the target storage node after the schema change operation is performed and the changed schema.

[0166] The query unit 830, in response to the target schema version identifier sent by the CDC, queries the schema corresponding to the target schema version identifier from the stored mapping relationship between each schema version identifier and the corresponding schema, and returns the queried schema to the CDC.

[0167] Optionally, the modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0168] Figure 11 An example is provided in the structural block diagram of a data processing apparatus corresponding to the aforementioned Figure 8The illustrated data processing method embodiment. Specifically, this data processing device can be applied to a target management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; and the device may include a receiving unit 910, a processing unit 920, and a writing unit 930.

[0169] The receiving unit 910 receives a database operation request distributed by the management node; wherein the database operation request includes a schema change operation for the distributed database.

[0170] Processing unit 920, in response to the schema change operation of the database operation request, updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation;

[0171] The writing unit 930 writes a schema change log into the write-ahead log; wherein the schema change log includes the updated schema version identifier.

[0172] Optionally, the database operation request further includes a data write operation, and the apparatus further includes:

[0173] The processing subunit, in response to the data write operation of the database operation request, writes the data to be written indicated by the data write operation into the distributed database, and generates a log corresponding to the data write operation in the write-ahead log accordingly.

[0174] Optionally, the modified schema version identifier includes a monotonically increasing schema version identifier based on the original schema version identifier.

[0175] Figure 12 An example is provided in the structural block diagram of a data processing apparatus corresponding to the aforementioned Figure 9The illustrated data processing method embodiment. Specifically, this data processing device can be applied to a CDC in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; and the device includes a reading unit 1010, a query unit 1020, and a consumption unit 1030.

[0176] Read unit 1010 reads the write-ahead log and asynchronously consumes executed database operation requests;

[0177] When query unit 1020 reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log.

[0178] Consumer unit 1030 processes the data writing operations corresponding to the logs between the schema change log and the next schema change log based on the queried schema.

[0179] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0180] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0181] like Figure 13 As shown, Figure 13 This specification illustrates a hardware structure diagram of an electronic device according to an embodiment of the present specification. The device may include: a processor 1710, a memory 1720, an input / output interface 1730, a communication interface 1740, and a bus 1750. The processor 1710, memory 1720, input / output interface 1730, and communication interface 1740 are internally connected to each other via the bus 1750.

[0182] The processor 1710 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above-described methods by running executable instructions.

[0183] The memory 1720 for storing processor-executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1720 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1720.

[0184] The input / output interface 1730 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0185] The communication interface 1740 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0186] Bus 1750 includes a pathway for transmitting information between various components of the device, such as processor 1710, memory 1720, input / output interface 1730, and communication interface 1740.

[0187] It should be noted that although the above-described device only shows the processor 1710, memory 1720, input / output interface 1730, communication interface 1740, and bus 1750, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0188] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method of any of the above embodiments.

[0189] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0190] This specification also provides a computer program product that, when executed by a processor, is used to implement the data processing method described in any embodiment of this specification.

[0191] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0192] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0193] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0194] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data processing method, characterized in that, The method is applied to a distributed storage system, which includes a distributed database, a write-ahead log (WAP) system, and a central deprecation control (CDC). The distributed database includes at least one management node and at least one storage node. The management node stores schema version identifiers for each schema version, with each schema version identifier corresponding to a schema after a schema change event is executed. The management node receives database operation requests initiated by the client and distributes the database operation requests to the target storage node; wherein, the database operation requests include schema change operations or data write operations for the distributed database; In response to the schema change operation of the database operation request, the target storage node updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation; further, it writes a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier; The CDC reads the write-ahead log, asynchronously consumes executed database operation requests, and when it reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log, and processes the data write operation corresponding to the log between the schema change log and the next schema change log based on the queried schema.

2. The method according to claim 1, characterized in that, Before the target storage node writes the schema change log to the write-ahead log, the process also includes: The target storage node reports the updated schema version identifier and the changed schema to the management node; The management node stores the correspondence between the updated schema version identifier and the changed schema.

3. The method according to claim 2, characterized in that, The step of querying the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log includes: The CDC sends the target schema version identifier recorded in the schema change log to the management node; The management node queries the schema corresponding to the target schema version identifier from the stored mapping between each schema version identifier and its corresponding schema, and returns the queried schema to the CDC.

4. The method according to claim 1, characterized in that, The modified schema version identifier is a monotonically increasing schema version identifier based on the original schema version identifier.

5. A data processing method, characterized in that, The method is applied to a management node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Censor); the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the method includes: Receive database operation requests initiated by clients; wherein, the database operation requests include schema change operations for distributed databases; The database operation request is distributed to the target storage node, and the correspondence between the updated schema version identifier returned by the target storage node after the schema change operation is performed and the changed schema is stored. In response to the target schema version identifier sent by the CDC, the schema corresponding to the target schema version identifier is queried from the stored mapping relationship between each schema version identifier and its corresponding schema, and the queried schema is returned to the CDC.

6. The method according to claim 5, characterized in that, The modified schema version identifier is a monotonically increasing schema version identifier based on the original schema version identifier.

7. A data processing method, characterized in that, The method is applied to a target storage node in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Catch-Up). The distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the method includes: Receive database operation requests distributed by the management node; wherein, the database operation requests include schema change operations for the distributed database; In response to the schema change operation of the database operation request, based on the schema change data indicated by the schema change operation, the schema of the distributed database is updated and an updated schema version identifier is generated; Write a schema change log to the write-ahead log; wherein the schema change log includes the updated schema version identifier.

8. The method according to claim 7, wherein the modified schema version identifier includes a schema version identifier that is monotonically incremented based on the schema version identifier before the modification.

9. A data processing method, characterized in that, A CDC (Cybersecurity Controller) is applied to a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and the CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores schema version identifiers for each schema version identifier of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the method includes: Read the write-ahead log and asynchronously consume executed database operation requests; When the schema change log is read, the management node is queried for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log. Based on the retrieved schema, perform data writing operations on the logs corresponding to the schema change logs between the schema change logs and the next schema change log.

10. A data processing apparatus, characterized in that, A management node applied in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Censor); the distributed database includes at least one of the management nodes and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the apparatus includes: The receiving unit receives database operation requests initiated by the client; wherein, the database operation requests include schema change operations for the distributed database. The storage unit distributes the database operation request to the target storage node and stores the correspondence between the updated schema version identifier returned by the target storage node after the schema change operation is performed and the changed schema. The query unit, in response to the target schema version identifier sent by the CDC, queries the schema corresponding to the target schema version identifier from the stored mapping relationship between each schema version identifier and its corresponding schema, and returns the queried schema to the CDC.

11. A data processing apparatus, characterized in that, A target management node applied in a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and a CDC (Distributed Data Censor); the distributed database includes at least one management node and at least one storage node; wherein, the management node stores various schema version identifiers of the distributed database, each schema version identifier corresponding to a schema after a schema change event is executed; the device includes: The receiving unit receives database operation requests distributed by the management node; wherein, the database operation requests include schema change operations for the distributed database. The processing unit, in response to the schema change operation of the database operation request, updates the schema of the distributed database and generates an updated schema version identifier based on the schema change data indicated by the schema change operation. The writing unit writes a schema change log into the write-ahead log; wherein the schema change log includes the updated schema version identifier.

12. A data processing apparatus, characterized in that, A CDC (Central Controller) is applied to a distributed storage system; the distributed storage system further includes a distributed database, a write-ahead log, and the CDC; the distributed database includes at least one management node and at least one storage node; wherein, the management node stores schema version identifiers for each schema version identifier of the distributed database, and each schema version identifier corresponds to a schema after a schema change event is executed; the apparatus includes: The read unit reads the write-ahead log and asynchronously consumes executed database operation requests; When the query unit reads the schema change log, it queries the management node for the schema corresponding to the target schema version identifier based on the target schema version identifier recorded in the schema change log. The consumption unit processes the data writing operations corresponding to the logs between the schema change log and the next schema change log based on the queried schema.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-9 by executing the executable instructions.

14. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-9.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-9.

Images