Data processing method and apparatus

Abstract

The application discloses a data processing method and device, which is used for acquiring the execution progress of DDL changes in each database sub-library in real time and simplifying the process of aggregating business data in each database sub-library. The scheme comprises the following steps: acquiring and analyzing target transaction change logs in multiple database sub-libraries, wherein the target transaction change log is a transaction change log with the minimum difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log; if the event type indicated by the target transaction change log is a data definition language (DDL) event, the information of the DDL event recorded in the target transaction change log is recorded in the target event slot corresponding to the DDL event.

Description

Technical Field

[0001] This application relates to the field of database partitioning technology, and in particular to a data processing method and apparatus. Background Technology

[0002] Currently, to achieve real-time performance and accuracy in Online Analytical Processing (OLAP) after database sharding, a "middleware synchronization model" is often used to aggregate multiple database shards into a unified data warehouse (also known as HIVE). Specifically, the synchronization middleware uses a pseudo-SLAVE protocol to obtain transaction change logs from the database server in real time, then synchronously parses the transaction change logs and converts them into HIVE SQL statements. This allows for the real-time aggregation of data from multiple database shards into HIVE, meeting the requirements for real-time data analysis.

[0003] However, in addition to changes in Data Manipulation Language (DML) statements, each database shard of an application master table also undergoes Data Definition Language (DDL) changes. Furthermore, a DDL change to an application master table typically needs to be executed across all database shards within that application. However, since the execution of DDL changes across all database shards of an application master table cannot be synchronized, and physically deployed database shards often cannot execute DDL changes simultaneously, the transaction change logs obtained by the synchronization middleware, which record DDL change information for each database shard of the application, also exhibit a chronological order.

[0004] This leads to inconsistencies in the table structures within a single application master table and within HIVE during a specific timeframe after a DDL change. In other words, some database shards have completed the DDL change, while others have not. Therefore, a further solution is needed to obtain real-time updates on the DDL change execution progress across all database shards of an application master table and to summarize this progress. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method and apparatus for real-time acquisition of the execution progress of DDL changes in various database sub-databases.

[0006] Firstly, a data processing method is provided, including:

[0007] Obtain and parse the target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log.

[0008] If the event type indicated by the target transaction change log is a Data Definition Language (DDL) event, then the information of the DDL event recorded in the target transaction change log is recorded in the target event slot corresponding to the DDL event.

[0009] Secondly, a data processing apparatus is provided, comprising:

[0010] The log acquisition module is used to acquire and parse target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log.

[0011] The event slot recording module is used to record the information of the DDL event recorded in the target transaction change log into the target event slot corresponding to the DDL event if the event type indicated by the target transaction change log is a Data Definition Language (DDL) event.

[0012] Thirdly, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method of the first aspect.

[0013] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the data processing method of the first aspect.

[0014] In this embodiment, target transaction change logs from multiple database shards are acquired and parsed. These target transaction change logs are those with the smallest difference between their generation time and current acquisition time; specifically, the latest transaction change log generated in each database shard is acquired as the target transaction change log. If the event type indicated by the target transaction change log is a Data Definition Language (DDL) event, the content of the DDL event recorded in the target transaction change log is recorded in the target event slot corresponding to the DDL event. Using this embodiment, whenever a new transaction change log containing a DDL event is generated in each database shard of the target application, the content of the DDL event recorded in that log is recorded in the target event slot corresponding to that DDL event. The progress of each database shard in processing the same DDL event can be intuitively obtained from the target event slot. That is, the database shard corresponding to the DDL event content recorded in the target event slot has completed the DDL event change, while the database shards that have not recorded relevant information in the target event slot have not completed the corresponding DDL event change. This allows for real-time monitoring of the progress of each database shard in executing the DDL event during a single DDL event change process. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0016] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of the present invention.

[0017] Figure 2 This is an example diagram illustrating a real-world scenario of the application of the data processing method provided in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of a data processing device provided in one embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The drawing numbers in this application are only used to distinguish the various steps in the solution and are not used to limit the execution order of the various steps. The specific execution order is as described in the specification.

[0020] As the amount of application data stored in relational databases continues to increase, many companies find that the data volume recorded in their relational databases has reached the recommended limit for the amount of data a single database table can record. In this case, horizontal or vertical database sharding can be used to distribute application data across multiple relational databases of the same type, thereby solving problems such as the capacity limitations, limited number of client connections, and performance degradation caused by storing data in a single database. Taking MySQL as an example, applications can use MySQL database sharding proxy middleware or MySQL database sharding DDL components to distribute their application data across multiple MySQL database shards. In this embodiment, the method is also applicable to other relational databases, such as Oracle.

[0021] However, when a DDL change is performed on multiple MySQL database shards within an application, each MySQL database shard needs to execute the corresponding change operation. During this process, because the execution times of the DDL changes differ across the MySQL database shards, the DDL change records aggregated to the application's synchronization middleware have a certain chronological order. Furthermore, the DDL change records aggregated to the application's HIVE via the synchronization middleware also have a chronological order. This means that within a given period of a single DDL change execution, some MySQL database shards may have already performed the change, while others may not have completed it. Therefore, how to obtain the real-time DDL change execution progress of each database shard in an application's master table and aggregate the execution progress of these database shards remains a pressing issue.

[0022] To address the aforementioned problems in the prior art, this application provides a data processing method. First, the concepts of database sharding, HIVE, DDL, and DML are introduced:

[0023] Database sharding is a method of storing data in a distributed manner. It mainly utilizes the business characteristics of data records to distribute and store data records across multiple databases using a certain algorithm.

[0024] HIVE: A data warehouse that encapsulates Hadoop's parallel computing tasks at a lower level, primarily used to process structured data on Hadoop. HIVE provides a SQL-like language for data querying and translates SQL statements into parallel computing tasks that run on Hadoop.

[0025] DDL stands for Data Definition Languages, which mainly defines various data objects (databases, tables, columns, indexes, etc.).

[0026] DML stands for Data Manipulation Language, which mainly consists of statements for adding, deleting, updating, and querying data records.

[0027] This application provides a data processing method applied to a data processing device. For example... Figure 1 As shown, the solutions provided in this application include:

[0028] S110, obtain and parse the target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log.

[0029] For a given application, each database shard corresponding to that application typically generates a target transaction change log. This means that each database shard usually generates a target transaction change log whenever a transaction change occurs for the target application. The current time of obtaining the target transaction change log specifically refers to the time when target transaction change logs from multiple database shards are retrieved. At the current time of retrieval, at least one target transaction change log must be retrieved; that is, at this time, at least one database shard has already generated a target transaction change log.

[0030] It should be noted that the multiple database shards described in S110 can jointly maintain a master table for the application. This master table is often quite large, therefore it can be maintained by multiple database shards. That is, each database shard maintains a portion of the table data in the master table. Whenever the master table undergoes changes, such as adding, modifying, or deleting columns, these multiple database shards often need to work together to complete the modification. Each database shard records the modification operation in its respective transaction change log.

[0031] In this context, each database shard corresponds to a target transaction change log (English name: BINLOG_EVENT, also known as a BINLOG event). That is, each database shard generates a transaction change log every time a transaction change (such as a DDL event) is executed. The target transaction change log mentioned in S110 is the most recently generated transaction change log when it is obtained, that is, the transaction change log with the smallest difference between its generation time and the current acquisition time.

[0032] This target transaction change log is used to record transaction changes during the execution of database transactions in a database sharded database. Each transaction change log consists of an event header and an event body. The event header may include the event occurrence time, event type, and event body length. The event body may include the event content; for example, for a DDL event, the event body may include the content of the DDL event, such as information about added columns, modified table information, and / or deleted columns.

[0033] In practical applications, to facilitate determining the order in which transaction change logs occur, a synchronization point can be assigned to each transaction change log to indicate its sequence number within the database shard. For example, synchronization points can be set to 1, 2, 3, ..., n. Transaction change log A at synchronization point 1 is the first transaction change log generated in database shard a, transaction change log B at synchronization point 2 is the second transaction change log generated in database shard a, and so on. Transaction change logs with later synchronization points are more recent in time.

[0034] For example, when summarizing the execution progress of DDL events in various database shards of an application, the transaction change log of the last synchronization point (i.e., the latest BINLOG event) in each database shard of the application can be obtained first as the target transaction change log.

[0035] Specifically, the byte streams of the BINLOG events from each database shard that recently received the application can be read. It should be understood that the most basic unit of data transmission during data transmission is a stream of bytes. In the embodiments of this specification, data transmission between the database shards and the synchronization middleware is often in binary form. The specific manifestation of the BINLOG event is a string of binary characters, which can be referred to as a byte stream during transmission.

[0036] As an example, the first 23 bytes of the BINLOG event byte stream can be read to parse out the BINLOG event header. The values ​​such as "event occurrence time, event type, and event body length" can be obtained from the BINLOG event header. Then, the byte that matches the "event body length" can be read from the 24th byte of the BINLOG event byte stream as the BINLOG event body, thus obtaining a complete BINLOG event, which includes the BINLOG event header and the BINLOG event body.

[0037] The event types in the BINLOG event header can include DDL events and DML events. As an example, the event type for a DDL event can be represented as "Query Event," which in practice would be "QUERY_EVENT." The event types for a DML event can be represented as "Write Column Event," "Update Column Event," and "Delete Column Event," which in practice would be "WRITE_ROWS_EVENT," "UPDATE_ROWS_EVENT," and "DELETE_ROWS_EVENT," respectively.

[0038] That is, if the event type obtained in the BINLOG event header is "QUERY_EVENT", the branch process of executing the DDL event can be entered; while if the event type obtained in the BINLOG event header is any one of "WRITE_ROWS_EVENT", "UPDATE_ROWS_EVENT" and "DELETE_ROWS_EVENT", the branch process of executing the DML event can be entered.

[0039] S120, if the event type indicated by the target transaction change log is a DDL event, then the information of the DDL event recorded in the target transaction change log is recorded in the target event slot corresponding to the DDL event.

[0040] Specifically, if the event type indicated in the event header of the target transaction change log is "QUERY_EVENT", it indicates that the event type indicated by the target transaction change log is a DDL event. At this time, the branch process for executing the DDL event can be entered.

[0041] Optionally, to save storage space in the target event slot and to quickly determine whether a DDL event has been recorded in the target event slot, this embodiment of the specification can obtain a summary of the change content of the DDL event in the target transaction change log, and record the summary of the change content of the DDL event, rather than the change content of the DDL event, in the target event slot. The summary of the change content of the DDL event can be calculated using a hash algorithm.

[0042] Thus, assuming the target event slot contains a summary of the changes to the DDL event, when recording the execution status of the DDL event in other database shards within the target event slot, the execution time of the DDL event can be directly recorded under the summary of the changes. Specifically, the target transaction change log includes: the table name of the DDL event, and the DDL event information including a summary of the changes, the database shard identifier of the DDL event, and the timestamp of the DDL event execution in the corresponding database shard.

[0043] Record the information of the DDL event recorded in the target transaction change log into the target event slot corresponding to the DDL event, including:

[0044] Determine the target event slot corresponding to the table name of the DDL event;

[0045] Record the summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event into the target event slot corresponding to the table name of the DDL event.

[0046] One master table is maintained by multiple database shards, and one master table corresponds to one event slot, meaning that the name of each master table can correspond to an event slot.

[0047] When the target event slot is not empty, it can maintain an "array of DDL event times for each database shard". This array contains the timestamps of the DDL events executed by each database shard. It should be understood that, to distinguish the timestamps of DDL events executed by each database shard, this array also includes the shard identifier of each database shard (e.g., the shard number; this shard number should be a unique identifier for each database shard).

[0048] As an example, the target event slot can be represented as "a summary of the changes to the DDL event, and an array of DDL event times for each database shard". The "array of DDL event times for each database shard" can be represented as [a1, a2, a3, ..., an]. The position identifiers 1 to n of a1 to an can correspond to the shard identifiers of each database shard, and the specific values ​​of a1 to an can be the timestamps of the DDL events executed by each database shard.

[0049] Optionally, since the application's master table typically corresponds to a target event slot, to simplify the information recorded in the target event slot and reduce the difficulty of extracting information from it, this embodiment of the specification can set the target event slot to two states: empty and non-empty. When the target event slot is empty, it indicates that the previous DDL event of the master table corresponding to the target event slot has been completed in all database shards, while the next DDL event has not yet started in the application's database shards. When the target event slot is non-empty, it indicates that the previous DDL event of the master table corresponding to the target event slot has not been fully completed in the application's database shards; that is, the DDL event has been completed in some database shards, but not in others.

[0050] Specifically, the summary of the changes made by the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event, including:

[0051] Determine if the target event slot is empty;

[0052] If the target event slot is empty, the summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event will be recorded in the target event slot corresponding to the table name of the DDL event.

[0053] If the target event slot is not empty, the timestamp of the DDL event and the database shard identifier of the DDL event will be recorded in the target event slot corresponding to the table name of the DDL event.

[0054] When the target event slot is empty, first create an empty "DDL event time array for each database shard". The position identifier of each element in this empty "DDL event time array for each database shard" can correspond to the identifier of each database shard. Then, when recording the timestamp of the DDL event into the target event slot corresponding to the table name of the DDL event, specifically, based on the identifier of each database shard, record the timestamp of the DDL event of each database shard into the corresponding element position in the "DDL event time array for each database shard" in the target event slot corresponding to the table name of the DDL event. At the same time, record a summary of the changes made to the DDL event in this target event slot.

[0055] When the target event slot is not empty, the timestamps of the DDL events of each database shard can be directly recorded in the corresponding element position of the "DDL event time array of each database shard" in the target event slot corresponding to the table name of the DDL event, based on the database shard identifier.

[0056] Optionally, since the target event slot only records the execution progress of a DDL event in each database shard of the application once, it is cleared after recording the execution timestamps of the DDL event in all database shards of the application. Therefore, to facilitate the determination of the DDL change trajectory of the application's master table, this embodiment of the specification, after synchronizing each DDL event to the data warehouse corresponding to the application, can also record the change content of each DDL event executed in each database shard of the application in the synchronization middleware. Specifically, the target transaction change log also includes: the change content of the DDL event, and the method further includes:

[0057] Synchronize the changes made in the DDL event to the data warehouse corresponding to the application; the DDL event corresponds to the application.

[0058] Acquire and record the changes made by the DDL event when the data shard corresponding to the application executes.

[0059] Specifically, the changes in DDL events are synchronized to the data warehouse corresponding to the application. These DDL events are the changes in DDL events executed in multiple database shards of the application.

[0060] The synchronization middleware can record two types of data metadata, including meta-${DB}-${tab} and meta-DDL-HIVE-${tab}.

[0061] The `meta-${DB}-${tab}` section records table metadata for each database shard in the application. It primarily stores metadata information for each database shard after each DDL event execution, including the timestamp of the DDL event, the changed column name, the changed column type, and the initial value. The information recorded in `meta-${DB}-${tab}` can be sorted in reverse chronological order by the timestamp of the DDL event.

[0062] The `meta-DDL-HIVE-${tab}` table records the change history of DDL events executed in the HIVE data warehouse, as well as information on DDL events executed by all database shards of the application, summarized by the target event slot. It primarily includes information such as the timestamp of the executed DDL event, the changes made to the DDL event, a summary of the changes, an array of DDL event times for each database shard, information on newly added columns, modified tables, and deleted columns. The information recorded in `meta-DDL-HIVE-${tab}` can be sorted in descending order of the timestamp of the executed DDL event.

[0063] The following is combined Figure 2The diagram illustrates a real-world scenario of the data processing method. This specification will detail the branching process of executing DDL events, specifically using one of the multiple applications maintained in the database as the target application, including:

[0064] S121, the synchronization middleware sequentially parses the database shard number, table name, and DDL statement from the latest target transaction change log sent from each database shard (database shard 1, database shard 2, ..., database shard n) of the target application. It then stores the changed table metadata after the DDL event in each database shard of the target application in the meta-${DB}-${tab} field of each database shard. This primarily includes the timestamp of the DDL event execution for database shard 1, database shard 2, ..., database shard n, as well as the table column names and column types from the changes made during the DDL event.

[0065] Among them, the latest target transaction change log sent in the target application's database shard 1, database shard 2, ..., database shard n are target transaction change log 1, target transaction change log 2, ..., target transaction change log n, respectively.

[0066] S122, the synchronization middleware calculates a summary of the changes in the DDL event based on the changes in the DDL event, and then finds the target event slot corresponding to the table name corresponding to the changes in the DDL event.

[0067] S123, if the target event slot is empty, write the "transaction change log reception time, summary of DDL event changes, and timestamp of DDL event execution" into the target event slot. After writing to the target event slot, write the information recorded in the target event slot into the meta-DDL-HIVE-${tab} record of the synchronization middleware. This mainly includes writing the "DDL event execution timestamp, summary of DDL event changes, DDL statement, information on newly added columns, information on modified tables, and information on deleted columns." Simultaneously, send the information recorded in the target event slot to the HIVE data warehouse corresponding to the target application, causing the HIVE data warehouse to execute the corresponding DDL event.

[0068] S124, if the target event slot is not empty, only add the timestamp of the DDL event of the currently received database shard to the "DDL event time array of each database shard" in the target event slot, and update the information recorded in the target event slot to the meta-DDL-HIVE-${tab} of the synchronization middleware record.

[0069] S125, when the target event slot contains information about all DDL events in the database shards, save all the information of the target event slot to the meta-DDL-HIVE-${tab} of the synchronization middleware record, and restore the target event slot to an empty state.

[0070] Optionally, in order to summarize the information of DDL events recorded in the target event slot, if the information of DDL events with the same table name is recorded in the corresponding target event slot, then the information of all DDL events in the target event slot is synchronized to the synchronization middleware of the target application corresponding to the DDL event.

[0071] Information about DDL events with the same table name is recorded in the corresponding target event slot. Specifically, after all database shards of the same application's master table have executed the same DDL event, the information of the DDL event executed by all database shards of the application's master table is recorded in the corresponding target event slot.

[0072] As an example, after the target event slot records the information of DDL events in all database shards corresponding to the target table name, the information of all DDL events in the target event slot corresponding to the target table name is updated to the meta-DDL-HIVE-${tab} of the target application's synchronization middleware.

[0073] Specifically, after recording the content of each database shard's DDL event, the content of the DDL event is first recorded in meta-${DB}-${tab}. Then, based on the summary of the DDL event's changes, the execution timestamp, and the shard identifier, it is written to the target event slot. Next, the DDL event written to the target event slot is executed in the HIVE data warehouse, and after successful execution, the changes to the executed DDL event in the HIVE data warehouse are recorded in meta-DDL-HIVE-${tab}. Finally, when the information of DDL events in database shards with the same table name as the DDL events is recorded in the target event slot, the information of the DDL events recorded in the target event slot is updated in the target application's synchronization middleware's meta-DDL-HIVE-${tab}.

[0074] Optionally, to facilitate recording the execution progress of the next DDL event for each database shard of the target application, and to save storage space in the target event slot, after synchronizing all DDL event information in the target event slot to the synchronization middleware of the target application when the target event slot records the content of all database shards executing the DDL event, the method provided in this embodiment further includes:

[0075] Clear the target event slot.

[0076] Furthermore, if the event type indicated in the parsed event header of the target transaction change log is any one of "WRITE_ROWS_EVENT", "UPDATE_ROWS_EVENT", or "DELETE_ROWS_EVENT", then the branch process for executing DML events is entered. Specifically, after obtaining and parsing the target transaction change logs from multiple database shards, the method provided in this specification embodiment further includes:

[0077] If the event type indicated by the target transaction change log is a DML event, then the target DDL event information is obtained from the target application's synchronization middleware. The target DDL event information is the information of the DDL event whose generation time is the smallest difference between the generation time of each DDL event information and the current acquisition time of the DML event.

[0078] Based on the information of the target DDL event, determine the first DDL event corresponding to the information that generated the target DDL event;

[0079] Based on the first DDL event, determine the timestamp sequence of the first DDL events generated by the database shards that execute the first DDL event in multiple database shards;

[0080] Adjust the DML statements sequentially according to the information of the first DDL event corresponding to the timestamp sequence.

[0081] The process involves retrieving target DDL event information from the target application's synchronization middleware. In practice, this can be done by extracting the "event occurrence time" from the event header of the target transaction change log as "event timestamp A," and parsing the DML statements in the event body to obtain the database sharding identifier and table name 'a'. Next, the table metadata is searched in the synchronization middleware's meta-${DB}-${tab} section with the query condition "meta-${DB}-${tab}.timestamp < 'event timestamp A'". The query results are then filtered, taking only the record with the largest "DDL event timestamp". After filtering, the metadata information for each column of the table at the corresponding time point is retrieved, and this largest "DDL event timestamp" is assigned to the variable 'DDL event timestamp B'.

[0082] Finally, in the `meta-DDL-HIVE-${tab}` section of the synchronization middleware, the DDL change trajectory array corresponding to table name 'a' is searched. The query condition is "meta-hive-${tab}.database-DDL event time == 0 or meta-hive-${tab}.database-DDL event time >= 'DDL event timestamp B'", which means checking whether a DDL event has occurred after DDL event timestamp B. If not, the DDL event corresponding to DDL event timestamp B is taken as the first DDL event. If so, the DDL event that occurred after event timestamp B is taken as the first DDL event. At this point, the information of the latest generated target DDL event can be obtained, that is, the information of the target DDL event is the DDL event with the smallest difference between the generation time of the information of each DDL event and the current acquisition time of the DDL event.

[0083] It should be understood that since the execution of DDL events in the various data shards of the target application has a certain chronological order, when adjusting DML statements, in order to maintain the consistency of table changes corresponding to the DDL event, adjustments should also be made sequentially according to the execution timestamps of the DDL events. Therefore, it is also necessary to determine the timestamp sequence of the first DDL event generated by the database shard executing the first DDL event in multiple database shards, and adjust the DML statements sequentially according to the information of the first DDL event corresponding to the timestamp sequence. Specifically, adjusting the DML statements sequentially according to the information of the first DDL event corresponding to the timestamp sequence includes:

[0084] The information of the first DDL event corresponding to the timestamp sequence is sorted according to the chronological order of the timestamp sequence, and the DML statements are adjusted accordingly based on the sorted information of the first DDL event.

[0085] The first DDL event includes at least one of adding a specified column to a table, deleting a specified column from a table, and changing a specified column from a table.

[0086] After adjusting the DML statements in sequence, the adjusted DML statements can be executed in the HIVE data warehouse. Obviously, DML statements adjusted in this way according to the execution timestamps of DDL events can be executed automatically in sequence, without the need for operations and maintenance personnel to maintain the execution order.

[0087] The following embodiments of this specification will describe in detail the branching process of executing DML events, including:

[0088] Step 141: Extract the "event timestamp A" of the DML event from the event header of the target transaction change log, and parse the event body of the target transaction change log to obtain the database sharding identifier and table name.

[0089] Step 142: Search for table metadata in the meta-${DB}-${tab} fields of each database shard recorded by the synchronization middleware. The query condition is "meta-${DB}-${tab}.timestamp < 'Event Timestamp A'". Then filter the query results, taking only the record with the largest "DDL event timestamp". After filtering, the metadata information of each column of the table at the corresponding time point is obtained, and the largest "DDL event timestamp" is assigned to the variable "DDL event timestamp B".

[0090] Step 143: In the meta-DDL-HIVE-${tab} record of the synchronization middleware, search for the change trajectory array of the DDL event corresponding to the table name in Step 141. The query condition is "meta-hive-${tab}.database-DDL event time == 0 or meta-hive-${tab}.database-DDL event time >= 'DDL event timestamp B'". That is, check whether a DDL event has occurred after the largest "DDL event timestamp B".

[0091] If no DDL event occurs after the largest "DDL event timestamp B" mentioned above, then the DDL event corresponding to DDL event timestamp B is taken as the preceding DDL event. If a DDL event occurs after the largest "DDL event timestamp B" mentioned above, then the DDL event that occurs after event timestamp B is taken as the preceding DDL event.

[0092] Step 144: Extract the change records of DDL events sequentially from the change trajectory array of DDL events in reverse order, and readjust each field of the DML statement accordingly.

[0093] If the change path contains an "ADD" column, add the corresponding new column in the DML statement; if the change path contains a "DROP" column, delete the corresponding column in the DML; if the change path contains a "CHANGE" column, modify the corresponding column in the DML.

[0094] Step 145: After adjusting the DML statements in sequence, the adjusted DML statements can be executed correctly in HIVE.

[0095] The solution employed in this invention allows for the recording of the content of a DDL event in each database shard of the target application as soon as a new transaction change log containing a DDL event is generated. This content is then recorded in the target event slot corresponding to the DDL event. The progress of each database shard in processing the same DDL event can be directly observed from this target event slot. Specifically, the database shard corresponding to the DDL event content recorded in the target event slot has completed the DDL event change, while database shards without such content in the target event slot have not completed the corresponding DDL event change. This enables real-time monitoring of the progress of each database shard in executing the DDL event during a single DDL event change process.

[0096] In order to solve the problems existing in the prior art, such as Figure 3 As shown in the illustration, this application also provides a data processing apparatus 300, comprising:

[0097] The log acquisition module 310 is used to parse and acquire target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log.

[0098] The event slot recording module 320 is used to record the information of the DDL event recorded in the target transaction change log into the target event slot corresponding to the DDL event if the event type indicated by the target transaction change log is a Data Definition Language (DDL) event.

[0099] The data processing apparatus provided in this application, employing the solution of this invention, enables the recording of the content of the DDL event in each database shard's transaction change log (which records a new DDL event) into the target event slot corresponding to that DDL event, whenever each database shard generates a new transaction change log containing a DDL event. From this target event slot, the progress of each database shard in processing the same DDL event can be intuitively obtained. That is, the database shard corresponding to the DDL event content recorded in the target event slot has completed the DDL event change, while the database shard that has not recorded relevant information in the target event slot has not completed the corresponding DDL event change. This allows for real-time monitoring of the progress information of each database shard executing the DDL event during a single DDL event change process.

[0100] Optionally, in one implementation, the target transaction change log includes: the table name of the DDL event; the information of the DDL event includes a summary of the changes made by the DDL event, the database shard identifier of the DDL event, and the timestamp of the database shard corresponding to the database shard identifier of the DDL event executing the DDL event; the event slot recording module 320 is used for:

[0101] Determine the target event slot corresponding to the table name of the DDL event;

[0102] The summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

[0103] Optionally, in one embodiment, the event slot recording module 320 is used to:

[0104] Determine whether the target event slot is empty;

[0105] If the target event slot is empty, then the summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

[0106] If the target event slot is not empty, then the timestamp of the DDL event and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

[0107] Optionally, in one implementation, the target transaction change log further includes: the change content of the DDL event, and the apparatus further includes:

[0108] The first synchronization module is used to synchronize the changes of the DDL event to the data warehouse corresponding to the application, wherein the DDL event corresponds to the application.

[0109] The recording module is used to acquire and record the changes made by the DDL event when the data sub-database corresponding to the application executes the record.

[0110] Optionally, in one embodiment, the device further includes:

[0111] The second synchronization module is used to synchronize the information of all DDL events in the corresponding target event slot to the synchronization middleware of the application corresponding to the DDL event when the information of the DDL events with the same table name is recorded in the corresponding target event slot.

[0112] Optionally, in one embodiment, the device further includes:

[0113] The clear module is used to clear the target event slot.

[0114] Optionally, in one embodiment, after the log acquisition module 410 acquires and parses the target transaction change logs in multiple database shards, the apparatus further includes:

[0115] The second acquisition module is used to acquire information about the target DDL event from the application's synchronization middleware if the event type indicated by the target transaction change log is a Data Manipulation Statement (DML) event. The information about the target DDL event is the information about the DDL event whose generation time is the smallest difference between the generation time of the information of each DDL event and the current acquisition time of the DML event.

[0116] The event determination module is used to determine, based on the information of the target DDL event, the first DDL event corresponding to the information that generated the target DDL event;

[0117] The sequence determination module is used to determine, based on the first DDL event, the timestamp sequence of the first DDL event generated by the database shard that executes the first DDL event among the multiple database shards;

[0118] The statement adjustment module is used to adjust DML statements sequentially according to the information of the first DDL event corresponding to the timestamp sequence.

[0119] In this application, the modules in the apparatus provided can also implement the method steps provided in the method embodiments. Alternatively, the apparatus provided in this application may include other modules besides those described above to implement the method steps provided in the method embodiments. Furthermore, the apparatus provided in this application can achieve the technical effects achievable by the method embodiments.

[0120] Preferably, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data processing method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0121] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data processing method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0122] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0128] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Obtain and parse the target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log. If the event type indicated by the target transaction change log is a Data Definition Language (DDL) event, the target transaction change log includes: the table name of the DDL event, and the information of the DDL event includes a summary of the changes in the DDL event, the database shard identifier of the DDL event, and the timestamp of the database shard corresponding to the database shard identifier of the DDL event executing the DDL event, then the target event slot corresponding to the table name of the DDL event is determined. The summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

2. The method as described in claim 1, characterized in that, The step of recording the summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event into the target event slot corresponding to the table name of the DDL event includes: Determine whether the target event slot is empty; If the target event slot is empty, then the summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event. If the target event slot is not empty, then the timestamp of the DDL event and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

3. The method as described in claim 2, characterized in that, The target transaction change log also includes: the change content of the DDL event, and the method further includes: The changes in the DDL event are synchronized to the data warehouse corresponding to the application, and the DDL event corresponds to the application. Acquire and record the changes made by the DDL event when the data shard corresponding to the application executes.

4. The method as described in claim 1, characterized in that, The method further includes: If the information of DDL events with the same table name is recorded in the corresponding target event slot, then the information of all DDL events in the corresponding target event slot will be synchronized to the synchronization middleware of the application corresponding to the DDL event.

5. The method as described in claim 4, characterized in that, The method further includes: Clear the target event slot.

6. The method as described in claim 1, characterized in that, After acquiring and parsing the target transaction change logs in multiple database shards, the method further includes: If the event type indicated by the target transaction change log is a Data Manipulation Statement (DML) event, then the information of the target DDL event is obtained from the application's synchronization middleware. The information of the target DDL event is the information of the DDL event whose generation time is the smallest difference between the generation time of each DDL event and the current acquisition time of the DML event. Based on the information of the target DDL event, determine the first DDL event corresponding to the information that generated the target DDL event; Based on the first DDL event, determine the timestamp sequence of the first DDL event generated by the database shard that executed the first DDL event in the plurality of database shards; Adjust the DML statements sequentially according to the information of the first DDL event corresponding to the timestamp sequence.

7. A data processing apparatus, characterized in that, include: The log acquisition module is used to acquire and parse target transaction change logs in multiple database shards. The target transaction change log is the transaction change log with the smallest difference between the generation time of each transaction change log and the current acquisition time of the target transaction change log. The event slot recording module is used to determine the target event slot corresponding to the table name of the DDL event if the event type indicated by the target transaction change log is a Data Definition Language (DDL) event, and the target transaction change log includes: the table name of the DDL event, the information of the DDL event including a summary of the change content of the DDL event, the database shard identifier of the DDL event, and the timestamp of the database shard corresponding to the database shard identifier of the DDL event executing the DDL event; The summary of the changes to the DDL event, the timestamp of the DDL event, and the database shard identifier of the DDL event are recorded in the target event slot corresponding to the table name of the DDL event.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

Images