Data processing methods, apparatus, electronic devices and computer-readable storage media

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and computer readable storage medium, and relate to the technical field of database. The method comprises: when an update operation log of an upstream database is obtained, parsing the update operation log, and adding the parsed update task into a preset update task set; performing at least one update operation on the update task set until a preset condition is met, to obtain a first task set after updating; generating an update statement based on the first task set, and writing the update statement into a downstream database, so that the downstream database and the upstream database are synchronously updated. The embodiments of the present application realize corresponding merging operations on update tasks according to real-time data types of each update task, reduce the read-write burden of the upstream database and the downstream database, avoid high latency during data synchronization, and improve data synchronization efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, in particular, the present application relates to a data processing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] The database is a warehouse that organizes, stores and manages data according to data structure, is a long-term storage in the computer, organized, sharable, unified management of a large number of data collection. Based on the consideration of technology upgrading or security needs, for example, when building a database cluster, there is often a need for data synchronization, that is, keeping the data real-time synchronization between two databases, in order to guarantee the consistency of source data (i.e. upstream database) and target data (i.e. downstream database).

[0003] In the prior art, the data update task is often synchronized in batches, that is, the data update task set is divided into batches according to the number of data update tasks, and the data update tasks are written into the downstream database in batches; when the data source is updated frequently, the number of data update tasks will increase, and the read-write burden of the upstream database and the downstream database in the above method is too large, there is a high delay when synchronizing data, resulting in low synchronization efficiency. SUMMARY

[0004] The embodiment of the present application provides a data processing method, device, electronic equipment and computer readable storage medium, which can solve the problem of low efficiency of data synchronization delay. The technical solution is as follows:

[0005] According to an aspect of the embodiment of the present application, a data processing method is provided, which comprises:

[0006] When the update operation log of the upstream database is obtained, the update operation log is parsed, and the parsed update task is added to the preset update task set;

[0007] At least one update operation is performed on the update task set until the preset condition is met, and a first task set after updating is obtained;

[0008] Based on the first task set, an update statement is generated, and the update statement is written into the downstream database, so that the downstream database is synchronized with the upstream database;

[0009] Wherein, the update operation comprises:

[0010] Determine the real-time data type of each update task in the update task set; wherein, the real-time data type is used to represent the update frequency of the data source corresponding to the update task;

[0011] According to the real-time data type of each update task, the corresponding update task is merged to obtain an update task set before a next update operation.

[0012] In a possible implementation, the above determining the real-time data type of each update task in the update task set comprises:

[0013] When the number of update tasks in the update task set is greater than a preset threshold, the update tasks are classified according to the data source corresponding to each update task to determine the real-time data type of each update task.

[0014] In a possible implementation, the above classifying the update tasks according to the data source corresponding to each update task to determine the real-time data type of each update task comprises:

[0015] Obtaining the historical update times of each data source in at least two preset time periods;

[0016] Based on the time period and the historical update times corresponding to the time period, the real-time data type of the data source is determined.

[0017] For each update task, the real-time data type of the corresponding data source is taken as the real-time data type of the update task.

[0018] In another possible implementation, the above classifying the update tasks according to the data source corresponding to each update task to determine the real-time data type of each update task comprises:

[0019] For each update task, the setting attribute of the data source corresponding to the update task is queried, and the real-time data type of the update task is determined according to the setting attribute.

[0020] In another possible implementation, the real-time data type comprises hot data and cold data; according to the real-time data type of each update task, the corresponding update task is merged to obtain an update task set before a next update operation, which comprises:

[0021] The update task corresponding to the hot data is processed to obtain at least one first update task.

[0022] The update task corresponding to the cold data is taken as a second update task.

[0023] According to the first update task and the second update task, an update task set before a next update operation is determined.

[0024] In yet another possible implementation, the above processing the update task corresponding to the hot data to obtain at least one first update task comprises:

[0025] determine a data source and an update operation type corresponding to each corresponding hot data update task;

[0026] For the update tasks of the same data source, the update operation types corresponding to each update task are compressed and merged to generate a first update task for the data source.

[0027] In yet another possible implementation, the preset condition includes at least one of the following:

[0028] The number of update operations reaches a preset number;

[0029] There is no update task corresponding to hot data in the update task set.

[0030] According to another aspect of the embodiments of the present application, a data processing apparatus is provided, which comprises:

[0031] The parsing module is configured to parse the update operation log when the update operation log of the upstream database is obtained, and add the parsed update task into a preset update task set;

[0032] The update module is configured to perform at least one update operation on the update task set until a preset condition is met, to obtain an updated first task set;

[0033] The update operation includes:

[0034] determine the real-time data type of each update task in the update task set; wherein the real-time data type is used to represent the update frequency of the data source corresponding to the update task; and merge the corresponding update tasks according to the real-time data type of each update task to obtain the update task set before the next update operation;

[0035] The synchronization module is configured to generate an update statement based on the first task set, and write the update statement into the downstream database, so as to synchronize the update of the downstream database with the upstream database.

[0036] In one possible implementation, when determining the real-time data type of each update task in the update task set, the update module is configured to:

[0037] When the number of update tasks in the update task set is greater than a preset threshold, the update tasks are classified according to the data source corresponding to each update task to determine the real-time data type of each update task.

[0038] In one possible implementation, when classifying the update tasks according to the data source corresponding to each update task to determine the real-time data type of each update task, the update module is configured to:

[0039] acquire a historical update frequency of each data source in at least two preset time periods;

[0040] determine the real-time data type of the data source based on the time period and the historical update frequency corresponding to the time period;

[0041] for each update task, take the real-time data type of the corresponding data source as the real-time data type of the update task.

[0042] In another possible implementation, when the update module classifies the update tasks according to the data sources corresponding to each update task and determines the real-time data type of each update task, it is configured to:

[0043] for each update task, query the setting attribute of the data source corresponding to the update task, and determine the real-time data type of the update task according to the setting attribute.

[0044] In another possible implementation, the real-time data type includes hot data and cold data; when the update module merges the corresponding update tasks according to the real-time data type of each update task to obtain the update task set before the next update operation, it is configured to:

[0045] merge the update tasks corresponding to the hot data to obtain at least one first update task;

[0046] take the update tasks corresponding to the cold data as second update tasks;

[0047] determine the update task set before the next update operation according to the first update task and the second update task.

[0048] In yet another possible implementation, when the update module merges the update tasks corresponding to the hot data to obtain at least one first update task, it is configured to:

[0049] determine the data source and the update operation type corresponding to each update task corresponding to the hot data;

[0050] for the update tasks of the same data source, compress and merge the update operation types corresponding to each update task to generate a first update task for the data source.

[0051] In yet another possible implementation, the preset condition includes at least one of the following:

[0052] the number of update operations reaches a preset number;

[0053] there is no update task corresponding to the hot data in the update task set.

[0054] According to another aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method shown in the first aspect of the embodiments of the present application.

[0055] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the method shown in the first aspect of the embodiments of the present application.

[0056] According to an aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, wherein the computer program is executed by a processor to implement the steps of the method shown in the first aspect of the embodiments of the present application.

[0057] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0058] The embodiments of the present application determine the update task by analyzing the update operation log, then perform at least one update operation on the update task set until the preset condition is met, to obtain the first task set after the update, and then generate the update statement based on the first task set, and write the update statement into the downstream database, so as to synchronize the update of the upstream database and the downstream database; wherein, in the update process of the update task set, the corresponding update task can be merged based on the real-time data type of each update task, since the real-time data type can represent the update frequency of the data source corresponding to the update task, so the update tasks can be targetedly merged according to the update frequency of the data source, to reduce the number of update tasks. Unlike the prior art of writing data update tasks in batches into the downstream database, the embodiments of the present application perform corresponding merging operation on the update tasks by determining the real-time data type of each update task, which reduces the read-write burden of the upstream database and the downstream database, avoids high latency during data synchronization, and improves the data synchronization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical schemes in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0060] Figure 1 An application scenario diagram of the data processing method provided by the embodiments of the present application;

[0061] Figure 2 A flowchart of the data processing method provided by the embodiments of the present application;

[0062] Figure 3 A flowchart of the update task classification in the data processing method provided by the embodiments of the present application;

[0063] Figure 4 A flowchart of a data processing method based on decision tree judgment provided for an embodiment of the present application is shown in FIG. 1.

[0064] Figure 5 A flowchart of an example data processing method provided for an embodiment of the present application is shown in FIG. 2.

[0065] Figure 6 A structural diagram of a data processing device provided for an embodiment of the present application is shown in FIG. 3.

[0066] Figure 7 A structural diagram of a data processing electronic device provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0067] Embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.

[0068] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the terms "comprise" and "include" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present art. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein means that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0069] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below in conjunction with the accompanying drawings.

[0070] A database, also known as a data management system, is an organized collection of structured information or data. The most common database currently stores data in a series of tables in the form of rows and columns, supporting users to conveniently access, manage, modify, update, control and organize data.

[0071] Database data synchronization refers to the process of using synchronization tools to instantly synchronize data generated by an upstream database to a downstream database. Database data synchronization tools are typically used when migrating and synchronizing data between different relational databases. To ensure real-time data synchronization between two data sources, these tools often provide a continuous synchronization task to guarantee data consistency between the data source and the target data.

[0072] The inventors discovered that in practical application scenarios, such as for order tables that need to be updated frequently, it is often necessary to manage the final status of orders. However, short-lived changes and updates in the order table often do not need to be synchronized to downstream databases. At the same time, frequent updates will bring significant synchronization delays, which will place a large read and write burden on upstream and downstream databases and reduce the data synchronization efficiency of the data source.

[0073] The data processing methods, apparatus, electronic devices, and computer-readable storage media provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0074] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0075] like Figure 1 As shown, the data processing method of this application can be applied to... Figure 1 In the scenario shown, specifically, when the server obtains the update operation log of the aforementioned database, it parses the update operation log and adds the parsed update tasks to a preset update task set. Then, the server performs at least one update operation on the update task set until the preset conditions are met, obtaining the updated first task set. Subsequently, an update statement is generated based on the first task set, and the update statement is written to the downstream database so that the upstream database and the downstream database are updated synchronously.

[0076] Figure 1 In the scenario shown, the above data processing method can be performed on the server; in other scenarios, it can also be performed on the terminal.

[0077] Those skilled in the art will understand that the “terminal” used herein can be a mobile phone, tablet computer, PDA (Personal Digital Assistant), MID (Mobile Internet Device), etc.; and the “server” can be implemented using a standalone server or a server cluster composed of multiple servers.

[0078] This application provides a data processing method, such as... Figure 2 As shown, this method, which can be applied to servers or terminals performing data processing, includes:

[0079] S201, When the update operation log of the upstream database is obtained, the update operation log is parsed and the parsed update task is added to the preset update task set.

[0080] The update operation log can be a binlog (binary log), which records all update events on the database and stores them on disk in binary form. It can be used to view the database change history, perform incremental backups and recovery, etc. Update tasks can include table structure change tasks, such as creating a table (CREATE TABLE) and modifying a table (ALTER TABLE), as well as table data modification tasks, such as inserting, updating, and deleting data.

[0081] Specifically, the server or terminal used for data processing can obtain the update operation log of the upstream database in real time based on a preset time interval. When the update operation log is obtained, it is parsed to obtain at least one update task, and then the update task is added to the preset update task set.

[0082] S202, perform at least one update operation on the updated task set until the preset conditions are met, and obtain the updated first task set.

[0083] Specifically, the server or terminal used for data processing can classify each update task, and then perform at least one update operation on the set of update tasks based on the type of each update task, until a preset condition is met, resulting in the updated first set of tasks. The preset condition can be that the number of update operations reaches a preset threshold.

[0084] Specifically, the update operation may include the following steps:

[0085] S2021, Determine the real-time data type of each update task in the update task set.

[0086] The aforementioned real-time data type can be determined based on the data source corresponding to the update task. Furthermore, the real-time data type can be used to represent the update frequency of the data source corresponding to the update task. For example, the real-time data type can include cold data and hot data, with the update frequency corresponding to hot data being higher than that corresponding to cold data.

[0087] Specifically, the server or terminal used for data processing can determine the data source corresponding to each update task, and then determine the real-time data type of the corresponding update task based on the update frequency of the data source.

[0088] S2022, merge the corresponding update tasks according to the real-time data type of each update task to obtain the set of update tasks before the next update operation.

[0089] Specifically, servers or terminals used for data processing can merge and compress update tasks that are hot data in real time, and use the merged and compressed update tasks and the corresponding cold data update tasks as the set of update tasks before the next update operation.

[0090] In this embodiment of the application, taking the update task including task A and task B as an example for specific explanation, task A can be adding data a and b to the data source, i.e., table 1, and task B can be deleting data a from table 1. When both task A and task B are hot data, task A and task B can be merged to obtain the merged task C, which adds data b to table 1. This embodiment of the application merges two update tasks into one task to reduce the number of update tasks and reduce the read and write load of the database.

[0091] S203, Generate update statements based on the first task set, and write the update statements to the downstream database so that the downstream database is updated synchronously with the upstream database.

[0092] The update statement can be an SQL (Structured Query Language) statement. SQL is a database query and programming language used to access data and query, update and manage relational database systems; SQL statements are a language for operating on the database.

[0093] Specifically, the server or terminal used for data processing can determine the query events for each data source based on the first task set, convert the query events into SQL statements, and write the SQL statements into the downstream database so that the downstream database can be updated synchronously with the upstream database, thus completing the synchronous migration of data.

[0094] This application embodiment determines update tasks by parsing update operation logs, then performs at least one update operation on the set of update tasks until preset conditions are met, resulting in an updated first task set. Update statements are then generated based on this first task set and written to the downstream database to ensure synchronized updates between the upstream and downstream databases. During the update process of the update task set, update tasks can be merged based on their real-time data type. Since the real-time data type represents the update frequency of the data source corresponding to each update task, targeted merging of update tasks can be performed according to the update frequency of the data source, reducing the number of update tasks. Unlike existing technologies that write data update tasks to the downstream database in batches, this application embodiment determines the real-time data type of each update task and performs corresponding merging operations, reducing the read / write burden on the upstream and downstream databases, avoiding high latency during data synchronization, and improving data synchronization efficiency.

[0095] This application provides a possible implementation method in which the real-time data type of each update task in the update task set is determined, including:

[0096] When the number of update tasks in the update task set exceeds a preset threshold, the update tasks are classified according to the data source corresponding to each update task, and the real-time data type of each update task is determined.

[0097] In this embodiment, each update task can be batch-processed according to the size of the update task set. Only when the number of update tasks in the update task set exceeds a preset threshold will the update tasks be classified and processed, so as to improve the efficiency of subsequent update operations of the update task set and lay a good foundation for data synchronization between upstream and downstream databases.

[0098] This application provides one possible implementation method, such as... Figure 3 As shown above, update tasks are categorized based on the data source corresponding to each update task, and the real-time data type of each update task is determined, including:

[0099] S301, obtain the number of historical updates for each data source within at least two preset time periods.

[0100] Specifically, the server or terminal used for data processing can determine at least two preset time periods based on the current timestamp, and count the number of historical updates for each data source within at least two preset time periods, as well as the time consumed for the preset number of updates.

[0101] In this embodiment of the application, taking the current timestamp as July 2nd 00:00 as an example, the server or terminal used for data processing can determine the first time period based on the current timestamp: the first time period is from 23:55 on July 1st to 00:00 on July 2nd within the first 5 minutes, the second time period is from 23:00 on July 1st to 00:00 on July 2nd within the first hour, and the third time period is from 00:00 on July 1st to 00:00 on July 2nd within the previous day. The number of times the data source is updated in the above three time periods can be determined. At the same time, the 10 latest updates before the time point of July 2nd 00:00 can be determined, and the difference between the earliest update time among the above 10 updates and the current timestamp is taken as the target update time.

[0102] S302 determines the real-time data type of the data source based on the time period and the historical update count corresponding to the time period.

[0103] Specifically, servers or terminals used for data processing can construct decision trees based on time periods and the number of historical updates corresponding to those time periods, and determine the real-time data type of the data source based on the decision trees.

[0104] In this embodiment, taking July 2nd 00:00 as an example, the server or terminal used for data processing can determine the first time period based on the current timestamp: the first 5 minutes (July 1st 23:55 to July 2nd 00:00), the second time period (July 1st 23:00 to July 2nd 00:00), and the third time period (July 1st 00:00 to July 2nd 00:00). It can also determine the number of times the data source is updated within these three time periods. Simultaneously, it can determine the 10 latest updates before July 2nd 00:00, and use the difference between the earliest update time among these 10 updates and the current timestamp as the target update time. Further, taking real-time data types including cold data and hot data as an example, a decision tree can be constructed based on the first time period, the second time period, the third time period, and the target update time, such as... Figure 4 As shown, the real-time data type is determined based on the decision tree described above.

[0105] S303: For each update task, the real-time data type of the corresponding data source is used as the real-time data type of the update task.

[0106] In this embodiment, the update frequency of the data source is determined based on the historical update count, update time, and current timestamp. Then, the real-time data type of the data source is determined based on the update frequency. This can effectively classify each update task, laying a good foundation for subsequent merging of update tasks and updating of the update task set, and further ensuring the accuracy and efficiency of data synchronization.

[0107] This application provides a possible implementation method in which the update tasks are classified according to the data source corresponding to each update task, and the real-time data type of each update task is determined, including:

[0108] For each update task, query the settings properties of the data source corresponding to the update task, and determine the real-time data type of the update task based on the settings properties.

[0109] Specifically, the server or terminal used for data processing can also query the setting attributes corresponding to the update task from the preset storage space for each update task to determine the real-time data type of the update task.

[0110] This setting attribute can be determined based on at least one of the following triggering operations:

[0111] The operation of dragging or moving the interface element corresponding to the real-time data type to the preset range of the preset user interface;

[0112] Click or touch operations on interface elements corresponding to real-time data types;

[0113] Input operations for the identifier of real-time data type in the preset input controls.

[0114] In this embodiment, users can set attributes of the data source through the above-mentioned triggering operation according to actual engineering applications, thereby determining the real-time data type of the update task corresponding to the data source. This realizes the setting of user-defined real-time data types, effectively facilitating database management and improving the flexibility of data synchronization.

[0115] This application provides a possible implementation method, in which the real-time data type mentioned above may include hot data and cold data.

[0116] Based on the real-time data type of each update task, the corresponding update tasks are merged to obtain the set of update tasks before the next update operation, including:

[0117] S401, merge the update tasks corresponding to the hot data to obtain at least one first update task.

[0118] Specifically, for a server or terminal used for data processing, the update tasks for the corresponding hot data can be merged according to the data source and update operation type of the update task to obtain at least one first update task.

[0119] S402, the update task corresponding to the cold data is taken as the second update task, and the set of update tasks before the next update operation is determined based on the first update task and the second update task.

[0120] In this embodiment, update tasks that belong to "hot" data can be merged, while update tasks that belong to "cold" data with low update frequency can be left unmerged. This reduces the number of update tasks in the update task set, thereby further reducing the latency of data synchronization between the upstream and downstream databases. The specific task merging process will be described in detail below.

[0121] This application provides a possible implementation method in which the update tasks corresponding to the heat data are merged to obtain at least one first update task, including:

[0122] S501, determine the data source and update operation type corresponding to each update task of corresponding hot data.

[0123] In this embodiment of the application, the update tasks corresponding to the hot data include Task A and Task B as an example for specific explanation. Task A can be adding data a and b to the data source, i.e., Table 1, and Task B can be deleting data a from Table 1. Therefore, the data source of Task A is Table 1, and the update operation type includes an INSERT event; the data source of Task B is Table 1, and the update operation type includes a DELETE event.

[0124] S502: For update tasks with the same data source, the update operation types corresponding to each update task are compressed and merged to generate the first update task for the data source.

[0125] In this embodiment, taking the update tasks corresponding to hot data, including Task A and Task B, as an example, Task A can add data a and b to the data source, i.e., Table 1, while Task B can delete data a from Table 1. Therefore, the data source for Task A is Table 1, and the update operation type includes an INSERT event; the data source for Task B is Table 1, and the update operation type includes a DELETE event. Since the data sources for both Task A and Task B include Table 1, Task A and Task B can be merged to obtain the merged Task C, which adds data b to Table 1. This embodiment merges two update tasks into one task to reduce the number of update tasks and alleviate the read / write load on the database.

[0126] This application merges update tasks for corresponding hot data, which can reduce the read and write load on upstream and downstream databases for frequently updated data sources, further compress the size of the update task set, and avoid large delays during database data synchronization.

[0127] This application provides a possible implementation method, wherein the above-mentioned preset conditions include at least one of the following:

[0128] (1) The number of update operations reaches the preset number; the preset number can be set to 2 times.

[0129] (2) There is no update task with corresponding hot data in the update task set.

[0130] In this embodiment, the number of update operations on the update task set can be limited by a preset number of times. The termination conditions of the update operations can also be limited by the real-time data type of the update tasks in the update task set. This effectively avoids data redundancy in the first task set after the update, ensures the accuracy of the update statement, and further improves the data synchronization efficiency of the upstream and downstream databases.

[0131] To better understand the data processing methods described above, the following section will combine... Figure 5 This paper details an example of a data processing method described in this application. This method can be applied to database synchronization components such as DataBus, an open-source, low-latency distributed database synchronization system that provides reliable data capture, transfer, and processing capabilities. The data processing method described above may include the following steps:

[0132] S601, when the database synchronization component obtains the update operation log of the upstream database, it parses the update operation log and adds the parsed update tasks to the preset update task set.

[0133] The update operation log can be binlog.

[0134] S602, when the number of update tasks in the update task set exceeds a preset threshold, the database synchronization component classifies the update tasks according to the data source corresponding to each update task and determines the real-time data type of each update task.

[0135] In this embodiment, update tasks can be batch-processed according to the size of the update task set. Only when the number of update tasks in the update task set exceeds a preset threshold will the update tasks be classified and processed, thereby improving the efficiency of subsequent update operations on the update task set.

[0136] S603, the database synchronization component obtains the number of historical updates for each data source within at least two preset time periods, and determines the real-time data type of the data source based on the time periods and the number of historical updates corresponding to the time periods.

[0137] The aforementioned real-time data types can include hot data and cold data.

[0138] Specifically, the database synchronization component can determine at least two preset time periods based on the current timestamp, and count the number of historical updates for each data source within these two preset time periods, as well as the time taken for each preset update. Then, a decision tree is constructed based on the at least two time periods, the corresponding historical update counts, and the time taken for each preset update, and the real-time data type of the data source is determined according to the decision tree.

[0139] S604, the database synchronization component uses the real-time data type of the corresponding data source as the real-time data type of the update task for each update task.

[0140] In this embodiment, the update frequency of the data source is determined based on the historical update count, update time, and current timestamp. Then, the real-time data type of the data source is determined based on the update frequency. This can effectively classify each update task, laying a good foundation for subsequent merging of update tasks and updating of the update task set.

[0141] S605, the database synchronization component merges the update tasks corresponding to the hot data to obtain at least one first update task.

[0142] Specifically, for update tasks corresponding to hot data, the database synchronization component can merge each update task according to the data source and update operation type corresponding to the update task to obtain at least one first update task.

[0143] In this embodiment, taking the update tasks corresponding to hot data, including Task A and Task B, as an example, Task A can add data a and b to the data source, i.e., Table 1, while Task B can delete data a from Table 1. Therefore, the data source for Task A is Table 1, and the update operation type includes an INSERT event; the data source for Task B is Table 1, and the update operation type includes a DELETE event. Since the data sources for both Task A and Task B include Table 1, Task A and Task B can be merged to obtain the merged Task C, which adds data b to Table 1. This embodiment merges two update tasks into one task to reduce the number of update tasks and alleviate the read / write load on the database.

[0144] S606, take the update task corresponding to the cold data as the second update task, and determine the set of update tasks before the next update operation based on the first update task and the second update task. Return to step S602 to perform the second update on the update task set, and continue to step S607 after the second update is completed.

[0145] S607: The updated task set is used as the first task set. An update statement is generated based on the first task set and written to the downstream database so that the downstream database is updated synchronously with the upstream database.

[0146] This application embodiment determines update tasks by parsing update operation logs, then performs at least one update operation on the set of update tasks until preset conditions are met, resulting in an updated first task set. Update statements are then generated based on this first task set and written to the downstream database to ensure synchronized updates between the upstream and downstream databases. During the update process of the update task set, update tasks can be merged based on their real-time data type. Since the real-time data type represents the update frequency of the data source corresponding to each update task, targeted merging of update tasks can be performed according to the update frequency of the data source, reducing the number of update tasks. Unlike existing technologies that write data update tasks to the downstream database in batches, this application embodiment determines the real-time data type of each update task and performs corresponding merging operations, reducing the read / write burden on the upstream and downstream databases, avoiding high latency during data synchronization, and improving data synchronization efficiency.

[0147] This application provides a data processing apparatus, such as... Figure 6 As shown, the data processing device 60 may include: a parsing module 601, an update module 602, and a synchronization module 603;

[0148] The parsing module 601 is used to parse the update operation logs of the upstream database when they are obtained, and add the parsed update tasks to the preset update task set.

[0149] Update module 602 is used to perform at least one update operation on the update task set until a preset condition is met, and obtain the updated first task set;

[0150] The update operations include:

[0151] Determine the real-time data type of each update task in the update task set; where the real-time data type is used to represent the update frequency of the data source corresponding to the update task; merge the corresponding update tasks according to the real-time data type of each update task to obtain the update task set before the next update operation.

[0152] Synchronization module 603 is used to generate update statements based on the first task set and write the update statements to the downstream database so that the downstream database is updated synchronously with the upstream database.

[0153] This application embodiment provides a possible implementation method in which the update module 602, when determining the real-time data type of each update task in the update task set, is used to:

[0154] When the number of update tasks in the update task set exceeds a preset threshold, the update tasks are classified according to the data source corresponding to each update task, and the real-time data type of each update task is determined.

[0155] This application embodiment provides a possible implementation method. When the update module 602 classifies the update tasks according to the data source corresponding to each update task and determines the real-time data type of each update task, it is used to:

[0156] Obtain the historical update count for each data source within at least two preset time periods;

[0157] Determine the real-time data type of the data source based on the time period and the historical update count corresponding to the time period;

[0158] For each update task, the real-time data type of the corresponding data source is used as the real-time data type of the update task.

[0159] This application embodiment provides a possible implementation method. When the update module 602 classifies the update tasks according to the data source corresponding to each update task and determines the real-time data type of each update task, it is used to:

[0160] For each update task, query the settings properties of the data source corresponding to the update task, and determine the real-time data type of the update task based on the settings properties.

[0161] This application embodiment provides a possible implementation, wherein the aforementioned real-time data types include hot data and cold data; when the update module 602 merges the corresponding update tasks according to the real-time data types of each update task to obtain the update task set before the next update operation, it is used to:

[0162] The update tasks corresponding to the hot data are merged to obtain at least one first update task;

[0163] The update task corresponding to the cold data is designated as the second update task.

[0164] The set of update tasks to be performed before the next update operation is determined based on the first and second update tasks.

[0165] This application embodiment provides a possible implementation method, wherein when the update module 602 merges the update tasks of the corresponding hot data to obtain at least one first update task, it is used to:

[0166] Determine the data source and update operation type corresponding to each update task for the corresponding hot data;

[0167] For update tasks that share the same data source, the update operation types corresponding to each update task are compressed and merged to generate the first update task for the data source.

[0168] This application provides a possible implementation method, wherein the above-mentioned preset conditions include at least one of the following:

[0169] The number of update operations has reached the preset number;

[0170] There is no update task for the corresponding hot data in the update task set.

[0171] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0172] This application embodiment determines update tasks by parsing update operation logs, then performs at least one update operation on the set of update tasks until preset conditions are met, resulting in an updated first task set. Update statements are then generated based on this first task set and written to the downstream database to ensure synchronized updates between the upstream and downstream databases. During the update process of the update task set, update tasks can be merged based on their real-time data type. Since the real-time data type represents the update frequency of the data source corresponding to each update task, targeted merging of update tasks can be performed according to the update frequency of the data source, reducing the number of update tasks. Unlike existing technologies that write data update tasks to the downstream database in batches, this application embodiment determines the real-time data type of each update task and performs corresponding merging operations, reducing the read / write burden on the upstream and downstream databases, avoiding high latency during data synchronization, and improving data synchronization efficiency.

[0173] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a data processing method. Compared with related technologies, this application can achieve the following: This application analyzes update operation logs to determine update tasks, then performs at least one update operation on the set of update tasks until preset conditions are met, obtaining an updated first task set. Update statements are then generated based on the first task set and written to a downstream database to synchronize the upstream and downstream databases. During the update process of the update task set, update tasks can be merged based on the real-time data type of each update task. Since the real-time data type represents the update frequency of the data source corresponding to the update task, each update task can be merged in a targeted manner according to the update frequency of the data source to reduce the number of update tasks. Unlike the prior art, which writes data update tasks to the downstream database in batches, this application determines the real-time data type of each update task and performs corresponding merging operations, reducing the read / write burden on the upstream and downstream databases, avoiding high latency during data synchronization, and improving data synchronization efficiency.

[0174] In one alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 700 includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the electronic device 700 may further include a transceiver 704, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 704 is not limited to one type, and the structure of this electronic device 700 does not constitute a limitation on the embodiments of this application.

[0175] Processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0176] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0177] The memory 703 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0178] The memory 703 stores computer programs that execute embodiments of this application, and the execution is controlled by the processor 701. The processor 701 executes the computer programs stored in the memory 703 to implement the steps shown in the foregoing method embodiments.

[0179] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, and tablets, as well as fixed terminals such as digital TVs and desktop computers.

[0180] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0181] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the following actions:

[0182] When the update operation log of the upstream database is obtained, the update operation log is parsed and the parsed update tasks are added to the preset update task set.

[0183] Perform at least one update operation on the updated task set until the preset conditions are met, and obtain the updated first task set;

[0184] An update statement is generated based on the first task set and written to the downstream database so that the downstream database is updated synchronously with the upstream database.

[0185] The update operations include:

[0186] Determine the real-time data type for each update task in the update task set; whereby the real-time data type is used to represent the update frequency of the data source corresponding to the update task.

[0187] By merging the corresponding update tasks according to the real-time data type of each update task, a set of update tasks before the next update operation is obtained.

[0188] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0189] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0190] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: When an update operation log of an upstream database is acquired, the update operation log is parsed, and an update task obtained by the parsing is added to a preset update task set; At least one update operation is performed on the update task set until a preset condition is met, and a first task set after the update is obtained; An update statement is generated based on the first task set, and the update statement is written into a downstream database, so that the downstream database is synchronously updated with the upstream database; The update operation comprises the following steps: The real-time data type of each update task in the update task set is determined; wherein the real-time data type is used to represent the update frequency of a data source corresponding to the update task; The corresponding update task is merged according to the real-time data type of each update task, and an update task set before the next update operation is obtained.

2. The method of claim 1, wherein, The determination of the real-time data type of each update task in the update task set comprises the following steps: When the number of update tasks in the update task set is greater than a preset threshold, the update tasks are classified according to the data source corresponding to each update task, and the real-time data type of each update task is determined.

3. The method of claim 2, wherein, The classification of the update tasks according to the data source corresponding to each update task and the determination of the real-time data type of each update task comprise the following steps: The historical update times of each data source in at least two preset time periods are acquired; The real-time data type of the data source is determined based on the time period and the historical update times corresponding to the time period; For each update task, the real-time data type of the corresponding data source is taken as the real-time data type of the update task.

4. The method of claim 2, wherein, The classification of the update tasks according to the data source corresponding to each update task and the determination of the real-time data type of each update task comprise the following steps: For each update task, the setting attribute of the data source corresponding to the update task is queried, and the real-time data type of the update task is determined according to the setting attribute.

5. The method of claim 1, wherein, The real-time data type comprises hot data and cold data; the merging of the corresponding update tasks according to the real-time data type of each update task to obtain an update task set before the next update operation comprises the following steps: The update tasks corresponding to the hot data are merged to obtain at least one first update task; The update tasks corresponding to the cold data are taken as second update tasks; The update task set before the next update operation is determined according to the first update task and the second update task.

6. The method of claim 5, wherein, The merging of the update tasks corresponding to the hot data to obtain at least one first update task comprises the following steps: The data source and the update operation type corresponding to each update task corresponding to the hot data are determined; For the update tasks of the same data source, the update operation types corresponding to each update task are compressed and merged to generate a first update task for the data source.

7. The method of claim 5, wherein, The preset condition comprises at least one of the following: The number of update operations reaches a preset number; There is no update task corresponding to hot data in the update task set.

8. A data processing apparatus, characterized by, The method comprises the following steps: The parsing module is configured to, when an update operation log of an upstream database is acquired, parse the update operation log, and add an update task parsed to a preset update task set; The updating module is configured to perform at least one update operation on the update task set until a preset condition is met, and obtain an updated first task set; The update operation includes: determining a real-time data type of each update task in the update task set, wherein the real-time data type is used to represent an update frequency of a data source corresponding to the update task, and merging corresponding update tasks according to the real-time data type of each update task to obtain an update task set before a next update operation; The synchronization module is configured to generate an update statement based on the first task set, write the update statement to a downstream database, and synchronize the downstream database with the upstream database for update.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images