Data synchronization method and device

Abstract

The invention discloses a data synchronization method and device, and the method comprises the steps: obtaining a data structure change event; constructing a coordination operator event based on the data change information, and issuing the coordination operator event to an operator in the data lake, so that the operator in the data lake obtains an updated data object constructed based on the data change information from the coordination operator event and stores the updated data object; setting a first mark bit of the operator in the data lake as a first mark when the period of storing the intermediate state is entered, and setting a second mark bit of the operator in the data lake as the first mark based on the first mark when the period of storing the intermediate state is entered; when the period of storing the intermediate state is entered, updating and writing the data into the client through the coordination operator based on the updated data object; and when the next period of storing the intermediate state is entered and the second flag bit of the operator except the coordination operator in the data lake is the first flag, the data synchronization efficiency can be improved by updating and writing the data into the client through the operator except the coordination operator based on the updated data object.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method and device. Background Art

[0002] As the amount of data continues to grow, the number of data analysis tasks also increases. The transactional database on the data source side is good at processing transactional transactions, but is not good at processing the correlation analysis between massive data. Therefore, the data in the transactional database on the data source side is usually synchronized to the data lake with massive data analysis capabilities.

[0003] At present, data lakes basically have the ability to analyze massive data, but as the analysis end has higher and higher requirements for the real-time performance of data in the data lake, real-time data lake technology has emerged. The real-time data lake can store business data from the data source, that is, the business data from the data source can be synchronized to the data lake in real time.

[0004] However, if the business data undergoes a structural change, the currently running processing process needs to be stopped and the framework restarted. When restarting, the write client will be initialized so that the write client updates the structural change event. In other words, the real-time data lake framework cannot dynamically process data structural change events, and the data reading process needs to be stopped manually and restarted, which reduces the efficiency of data synchronization. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a data synchronization method and device, aiming to improve data synchronization efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, the method comprising:

[0007] Obtaining a data structure change event; the data structure change event includes data change information;

[0008] Constructing a coordination operator event based on the data change information, and sending the coordination operator event to operators in the data lake, so that the operators in the data lake obtain and store updated data objects constructed based on the data change information from the coordination operator event;

[0009] Setting a first mark bit of the operator in the data lake to a first mark, and when entering a cycle of saving an intermediate state, setting a second mark bit of the operator in the data lake to the first mark based on the first mark;

[0010] When entering the cycle of saving the intermediate state, based on the updated data object, the update is written to the client through the coordination operator;

[0011] When entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, based on the updated data object, the write client is updated by the operator other than the coordination operator.

[0012] Optionally, the data lake includes parallel processing operators, and the acquiring of data structure change events further includes:

[0013] Determining a key value in the data structure change event based on a target parallelism of an operator processed in parallel in the data lake; the target parallelism is used to indicate the number of units processed in parallel in the operator processed in parallel;

[0014] The constructing a coordination operator event based on the data change information and sending the coordination operator event to an operator in the data lake includes:

[0015] A coordination operator event is constructed based on the key value and the data change information, and the coordination operator event is sent to the operator in the data lake.

[0016] Optionally, determining the key value in the data structure change event based on the parallelism number of the operator processed in parallel in the data lake includes:

[0017] Calculating the maximum parallelism of the operators in parallel processing according to the target parallelism of the operators in parallel processing;

[0018] Calculating the maximum random value of the operators processed in parallel according to the target parallelism;

[0019] Calculation is performed according to the maximum parallelism, the maximum random value, and the target parallelism to determine key values ​​corresponding to the parallelism numbers of operators processed in parallel in the data lake.

[0020] Optionally, the calculating according to the maximum parallelism, the maximum random value and the target parallelism to determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake, respectively, includes:

[0021] By allocating a parallel degree number processing method, calculation is performed based on the maximum parallel degree, the maximum random value and the target parallel degree, and key values ​​corresponding to the parallel degree numbers of operators processed in parallel in the data lake are determined.

[0022] Optionally, constructing a coordination operator event based on the key value and the data change information, and sending the coordination operator event to an operator in the data lake includes:

[0023] constructing a coordination operator event based on the key value and the data change information;

[0024] Performing hash calculation on the key value to determine the parallelism allocation result of the coordination operator event;

[0025] According to the parallelism allocation result, the coordination operator event is sent to the operator in the data lake.

[0026] Optionally, when entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, based on the updated data object, the write client is updated by the operator other than the coordination operator, and then further includes:

[0027] The second mark bit of the operators in the data lake except the coordination operator is adjusted to the second mark.

[0028] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0029] An acquisition module, used to acquire a data structure change event; the data structure change event includes data change information;

[0030] A construction module, configured to construct a coordination operator event based on the data change information, and send the coordination operator event to the operator in the data lake, so that the operator in the data lake obtains and stores the updated data object constructed based on the data change information from the coordination operator event;

[0031] A setting module, configured to set a first mark bit of the operator in the data lake to a first mark, and when entering a cycle of saving an intermediate state, set a second mark bit of the operator in the data lake to the first mark based on the first mark;

[0032] A first update module, configured to update and write to the client through a coordination operator based on the update data object when entering a cycle of saving the intermediate state;

[0033] The second update module is used to update the write client through the operator other than the coordination operator based on the updated data object when entering the next cycle of saving the intermediate state and when the second mark bit of the operator other than the coordination operator in the data lake is the first mark.

[0034] Optionally, the data lake includes operators for parallel processing, and the device further includes:

[0035] A determination unit, configured to determine a key value in the data structure change event based on a target parallelism of an operator processed in parallel in the data lake; the target parallelism is used to indicate the number of units processed in parallel in the operator processed in parallel;

[0036] Correspondingly, the construction module includes:

[0037] The first construction unit is used to construct a coordination operator event based on the key value and the data change information, and send the coordination operator event to the operator in the data lake.

[0038] Optionally, the determining unit includes:

[0039] A first calculation unit, configured to calculate a maximum degree of parallelism of the operators in parallel processing according to a target degree of parallelism of the operators in parallel processing;

[0040] A second calculation unit, used for calculating the maximum random value of the operator processed in parallel according to the target parallelism;

[0041] The third calculation unit is used to perform calculation according to the maximum parallelism, the maximum random value and the target parallelism, and determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake.

[0042] Optionally, the third computing unit includes:

[0043] The fourth computing unit is configured to determine key values ​​corresponding to the parallelism numbers of operators processed in parallel in the data lake by performing calculations based on the maximum parallelism, the maximum random value, and the target parallelism by assigning a parallelism number processing method.

[0044] Optionally, the first construction unit includes:

[0045] A second construction unit, configured to construct a coordination operator event based on the key value and the data change information;

[0046] A hash calculation unit, used to perform hash calculation on the key value to determine the parallelism allocation result of the coordination operator event;

[0047] The sending unit is used to send the coordination operator event to the operator in the data lake according to the parallelism allocation result.

[0048] Optionally, the device further comprises:

[0049] An adjustment unit is used to adjust the second mark bit of the operator other than the coordination operator in the data lake to the second mark.

[0050] In a third aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0051] Memory for storing computer programs;

[0052] The processor is used to execute the computer program so that the device executes the data synchronization method described in the first aspect.

[0053] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein a computer program is stored on the computer-readable storage medium. When the computer program is executed, a device executing the computer program implements the data synchronization method described in the first aspect above.

[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0055] The embodiment of the present application provides a data synchronization method and device, the method comprising: obtaining a data structure change event including data change information, that is, first obtaining a data structure change event in which a data structure change occurs. Then, constructing a coordination operator event based on the data change information, and sending the coordination operator event to the operator in the data lake, so that the operator in the data lake obtains and stores the updated data object constructed based on the data change information from the coordination operator event. Afterwards, the first mark bit of the operator in the data lake is set to the first mark to mark that the operator in the data lake has stored the updated data object constructed based on the data change information, so that the operator in the data lake can be updated according to the stored updated data object containing the data change information. When entering the cycle of saving the intermediate state, the second mark bit of the operator in the data lake is set to the first mark based on the first mark to mark that the operator in the data lake has met the update conditions and is waiting for the update opportunity. And based on the updated data object, the client can be updated and written through the coordination operator. Since the coordination operator is used to enable a new processing process, when the processing process enters the cycle of saving the intermediate state, the client can be updated and written based on the coordination operator to ensure the accuracy of the update timing. Finally, when entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, the write client can be updated based on the updated data object through the operator other than the coordination operator in the data lake. It can be seen that when the processing process enters the next cycle of saving the intermediate state, that is, after completing the update of the coordination operator to write to the client, the update of other operators to write to the client is completed based on the updated data object. The above method can ensure the timing accuracy of updating the data object when the data structure changes, thereby completing the real-time update of the data structure change without stopping and restarting, thereby improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A schematic diagram of an application scenario of a data synchronization method provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the structure of data processing provided in an embodiment of the present application;

[0060] Figure 4 A timing diagram of data synchronization provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a structure for pre-calculating parallelism provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] Change data capture (CDC) is a real-time extraction-transformation-load (ETL) method, which is mainly used in data integration and analysis related fields during enterprise information planning and implementation.

[0065] As the amount of data continues to grow, the number of data analysis tasks also increases. The transactional database on the data source side is good at processing transactional transactions, but is not good at processing the correlation analysis between massive data. Therefore, the data in the transactional database on the data source side is usually synchronized to the data lake with the ability to analyze massive data. The data lake refers to a system that stores various raw data and has the ability to store, calculate, and query.

[0066] Traditional data lakes are based on the open source distributed file system Hadoop Distributed File System (HDFS) as the storage base and the data warehouse Hive as the computing query engine, which enables the data lake to basically have the ability to analyze massive data. However, as the analysis end has higher and higher requirements for the real-time performance of the data in the data lake, such as real-time statistics of hospital admissions and discharges, ward visits, and real-time surgery in the medical field, real-time data lake technology has emerged. The real-time data lake can store business data from the data source, that is, the business data from the data source can be synchronized to the data lake in real time.

[0067] At present, various industries usually use the data lake engine Hudi as a real-time data lake engine solution. Hudi has good update and deletion performance and rich query capabilities, and has a high community activity. Flink is a streaming computing engine. In the solution of real-time CDC data into the lake, it usually forms a Flink+Hudi streaming data ingestion framework with the Hudi framework. It has been widely used in many fields, such as e-commerce, logistics, and finance.

[0068] In the process of real-time data entry into the lake, in addition to synchronizing the data from the data source to the data lake in real time, the data modification language (DML) operations will also be synchronized to the data lake in real time. The data will generate avro files and parquet files in the storage format of big data files under the Hudi framework, which include data content. In addition, if the data structure in the data source has changed, such as data stored in a table structure, the table structure has changed, such as adding a column of data or reducing a column of data in the table structure, the data structure change should also be updated to the data lake. In the Flink+Hudi native framework, data structures such as table structures are synchronized only at the beginning of initialization. Data definition language (DDL) usually refers to table structure changes that cannot be synchronized during operation, resulting in the table structure in the data lake being inconsistent with the data source (i.e., the business source), thereby affecting data analysis in the data lake.

[0069] However, if the business data has a structural change, the currently running processing process needs to be stopped, that is, the processing process running under the Flink+Hudi framework needs to be stopped and restarted. When restarting, the write client writeClient will be initialized so that the write client updates the structural change event, that is, the written avro file and parquet file contain the changed data structure. In other words, the real-time data lake framework cannot dynamically process the data structure and change events, and the data reading process needs to be stopped manually and restarted, which reduces the efficiency of data synchronization.

[0070] Based on this, in order to solve the above problem, in an embodiment of the present application, a data structure change event including data change information is obtained, that is, a data structure change event in which a data structure change occurs is first obtained. Then, a coordination operator event is constructed based on the data change information, and the coordination operator event is sent to the operator in the data lake, so that the operator in the data lake obtains and stores the updated data object constructed based on the data change information from the coordination operator event. After that, the first mark bit of the operator in the data lake is set to the first mark to mark that the operator in the data lake has stored the updated data object constructed based on the data change information, so that the operator in the data lake can be updated according to the stored updated data object containing the data change information. When entering the cycle of saving the intermediate state, the second mark bit of the operator in the data lake is set to the first mark based on the first mark to mark that the operator in the data lake has met the update conditions and is waiting for the update opportunity. And based on the updated data object, the client can be updated and written through the coordination operator. Since the coordination operator is used to enable a new processing process, when the processing process enters the cycle of saving the intermediate state, the client can be updated based on the coordination operator to ensure the accuracy of the update timing. Finally, when entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, the write client can be updated based on the updated data object through the operator other than the coordination operator in the data lake. It can be seen that when the processing process enters the next cycle of saving the intermediate state, that is, after completing the update of the coordination operator to write to the client, the update of other operators to write to the client is completed based on the updated data object. The above method can ensure the timing accuracy of updating the data object when the data structure changes, thereby completing the real-time update of the data structure change without stopping and restarting, thereby improving the efficiency of data synchronization.

[0071] For example, one of the scenarios of the embodiment of the present application may be applied to Figure 1In the scenario shown. The scenario includes a database 101 and a server 102, wherein the database 101 includes a data structure change event, and the server 102 adopts the implementation method provided by the embodiment of the present application to obtain the data structure change event including data change information from the database 101 to perform data synchronization based on the data structure change event.

[0072] First, in the above application scenario, although the action description of the implementation method provided in the embodiment of the present application is executed by the server 102; however, the embodiment of the present application is not limited in terms of the execution subject, as long as the actions disclosed in the implementation method provided in the embodiment of the present application are executed.

[0073] Secondly, the above scenario is only an example scenario provided by the embodiment of the present application, and the embodiment of the present application is not limited to this scenario.

[0074] The specific implementation of the data synchronization method and device in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0075] See also Figure 2 , which is a flow chart of a data synchronization method provided in an embodiment of the present application, combined with Figure 2 As shown, it may specifically include:

[0076] S201: Acquire a data structure change event, where the data structure change event includes data change information.

[0077] Data change information refers to data information that has changed in the data source end, for example, data information in the table structure that has changed in the business source end that provides data, such as the data information in a newly added column, etc. Alternatively, data change information can also be metadata information that has changed in the table structure, such as a newly added row of data in the table structure, etc. In this regard, this application does not specifically limit data change information. A data structure change event refers to an event in which the data structure has changed, such as the data in the data source end is stored in the form of a table structure, a new column of data is added or a column of data is reduced in the table structure, etc. A data structure change event includes data change information, such as a data structure change event includes data information in the changed table structure.

[0078] S202: Construct a coordination operator event based on the data change information, and send the coordination operator event to the operators in the data lake, so that the operators in the data lake obtain and store the updated data objects constructed based on the data change information from the coordination operator event.

[0079] The overall data processing sequence in the Flink-Hudi framework is as follows Figure 3 As shown, Figure 3A structural diagram of data processing provided for an embodiment of the present application, firstly, source log mining is performed on the data source end Source, and the data to be processed (i.e., the data change information that has changed) is mined from the business log of the data source end, and then the data can be deserialized and converted through the deserialization conversion operator, and the format conversion operator is used to convert the deserialized data into a data format that can be circulated in the Flink framework. After that, the data indexing stage is entered, and the data indexing operator is used to locate the corresponding storage location in the HDFS system according to the key value Key of the data, and the data change information is written into the HDFS file through the stream data writing operator. After the data is written, the coordinator can be notified to synchronize the data to Hive through the coordination operator. At the same time, an event notification can be sent to the downstream operator to generate a compression merge plan, and the result submission is completed after the log is compressed by the log compression operator.

[0080] Among them, the above-mentioned processing processes need to be executed in a processing order to ensure the accuracy of the timing, with a high degree of timing function and sequential cohesion. The above-mentioned deserialization conversion operator is used to convert the serialized data (such as byte sequence or string form) back to its original object or data structure; the format conversion operator is used to convert the deserialized data into a data format that can be circulated in the Flink framework; the data index operator is used to locate the storage location of the data in the HDFS system; the stream data writing operator is used to write data change information to the HDFS file; the coordination operator is used to start a new processing process timeline server, and is used to synchronize the updated data object to Hive; the log compression operator is used to reduce the storage space occupied by the log.

[0081] After the deserialization operator obtains the data structure change event, it can construct a new complete updated data object newConfig based on the data change information in the data structure change event, and then construct a coordination operator event CoordinatorEvent (CEvent), which includes the updated data object. The coordination operator event can then be sent to downstream operators so that downstream operators in the data lake can obtain and store the updated data object constructed based on the data change information from the coordination operator event.

[0082] Updated data objects refer to objects that are reconstructed based on data change information and are applicable to updating data structures in the data lake. Coordination operator events refer to events used to notify the coordinator. The coordinator is used in the data lake to receive query requests from clients, decompose these requests into a series of tasks, and then assign them to operators in the data lake for processing.

[0083] S203: Set the first mark bit of the operator in the data lake to the first mark, and when entering the cycle of saving the intermediate state, set the second mark bit of the operator in the data lake to the first mark based on the first mark.

[0084] The first flag of an operator refers to the flag inside the operator in the data lake, which is used to indicate whether the operator has received a coordination operator event to retrieve the updated data object from the coordination operator event; the first flag is used to indicate that the operator has received a coordination operator event, that is, a data structure change has occurred. For example, the first flag of an operator can be represented by hasDDL to indicate whether a data structure change has occurred, and the first flag can be represented by true. When hasDDL is true, it indicates that the operator has received a coordination operator event and can retrieve the updated data object from the coordination operator event. Conversely, when hasDDL is false, it indicates that the operator has not received a coordination operator event and cannot retrieve the updated data object from the coordination operator event, that is, no data structure change has occurred. In addition, hasDDL is false by default.

[0085] As an example, see Figure 4 , Figure 4 A timing diagram of data synchronization provided in an embodiment of the present application, when a data index operator receives a coordination operator event, the internal flag bit hasDDL of the operator can be set to true, so that the data index operator can take out the updated data object newConfig from the coordination operator event and save it in the local attribute, while continuing to send the coordination operator event to the downstream;

[0086] When the stream data writing operator receives the coordination operator event, the internal flag bit hasDDL of the operator can be set to true, so that the stream data writing operator can extract the updated data object newConfig from the coordination operator event and save it in the local attribute, while continuing to send the coordination operator event to the downstream;

[0087] When the coordination operator receives a coordination operator event, the internal flag bit hasDDL of the operator can be set to true.

[0088] When the log compression operator receives the coordination operator event, the operator's internal flag hasDDL can be set to true, so that the log compression operator can retrieve the updated data object newConfig from the coordination operator event and save it in the local attribute.

[0089] After the above-mentioned response processing steps of different operators to the coordination operator event, the respective marking bits and attribute information are updated, so that each operator stores the updated data object.

[0090] Then, when entering the periodic checkpoint for saving the intermediate state, when the first mark bit of the operator in the data lake is the first mark, the second mark of the operator can be set to the first mark of the device.

[0091] In the Flink-Hudi framework, checkpoint is used to save the intermediate state of the job during stream processing. The state information of the Flink task can be periodically saved to a persistent storage medium so that the execution state of the task can be restored in time when a task fails, thereby achieving fault tolerance for data structure update tasks.

[0092] The second flag of the operator refers to the flag of the operator content in the data lake, which is used to indicate whether the coordination operator has completed the update, that is, when entering the cycle of saving the intermediate state, the coordination operator will complete the update based on the updated data object in advance, and other operators will complete the update after the coordination operator is updated. Since the coordination operator is used to start a new processing process in the data lake, other operators in the data lake access the processing process. Only when the coordination operator is updated and then other operators are updated can the timing accuracy of the data update be guaranteed. When the second flag is the first flag, it is used to indicate that the operator has met the update conditions and only needs to wait for the update opportunity, that is, the operator has stored the updated data object, and can be updated after the coordination operator completes the update. For example, the second flag of the operator can be represented by whether the coordination operator has completed the update isReady, and the first flag can be represented by true. When isReady is true, it means that the coordination operator has completed the update in this cycle, and the operator has met the update conditions. In the next cycle, the operator can be updated. On the contrary, when isReady is false, it means that the coordination operator has not completed the update in this cycle, and the operator has the update conditions, but does not have the update opportunity. In addition, isReady is false by default.

[0093] As an example, when hasDDL of the data index operator is true, when entering the cycle of saving the intermediate state, such as when the data index operator enters the i-th checkpoint, the internal flag bit isReady of the operator can be set to true;

[0094] When hasDDL of the stream data writing operator is true, when entering the cycle of saving the intermediate state, such as when the stream data writing operator enters the i-th checkpoint, the internal flag bit isReady of the operator can be set to true;

[0095] When entering the cycle of saving intermediate states, if the coordination operator enters the i-th checkpoint, the update is written to the client through the coordination operator.

[0096] When hasDDL of the log compression operator (log compression operator) is true, when entering the cycle of saving the intermediate state, such as the log compression operator entering the i-th checkpoint, the internal flag bit isReady of the operator can be set to true.

[0097] S204: When entering the cycle of saving the intermediate state, based on updating the data object, the update is written to the client through the coordination operator.

[0098] When entering the periodic checkpoint for saving intermediate states, the coordination operator can update the write client based on the updated data object, so that the write client can be updated based on the updated data object during the coordination operator phase.

[0099] The write client is used to indicate the program processing object responsible for writing data, that is, the client responsible for writing updated data objects to avro files and parquet files.

[0100] S205: When entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, based on the updated data object, the update is written to the client through the operator other than the coordination operator.

[0101] When entering the next cycle of saving the intermediate state, the second mark bit of other operators in the data lake except the coordination operator can be judged. If the second mark bit of the operator in the data lake except the coordination operator is the first mark, it means that other operators in the data lake can update and write to the client according to the stored updated data object.

[0102] As an example, when the data index operator enters the next cycle of saving the intermediate state, such as when the data index operator enters the i+1th checkpoint, it is determined whether isReady is true. If it is true, the data index operator can be used to update the write client based on the updated data object stored by the operator, that is, the client can be updated based on the stored updated data object during the data indexing phase. At the same time, after the update is completed, the second mark bit can also be adjusted to the second mark, such as adjusting isReady to false. The second mark is used to indicate that the update is not completed, so as to facilitate the subsequent data synchronization update based on the new data change information, which helps to improve data synchronization efficiency. In addition, after the update is completed, the first mark bit can also be adjusted to the second mark, such as adjusting hasDDL to false.

[0103] When the stream data write operator enters the next cycle of saving the intermediate state, such as when the stream data write operator enters the i+1th checkpoint, it is determined whether isReady is true. If it is true, the stream data write operator can be used to update the write client based on the updated data object stored by the operator, that is, the write client can be updated based on the stored updated data object during the stream data write phase. At the same time, after the update is completed, the second mark bit can also be adjusted to the second mark, such as adjusting isReady to false. In addition, after the update is completed, the first mark bit can also be adjusted to the second mark, such as adjusting hasDDL to false.

[0104] When the log compression operator enters the next cycle of saving the intermediate state, such as when the log compression operator enters the i+1th checkpoint, it is determined whether isReady is true. If it is true, the log compression operator can be used to update the client based on the updated data object stored by the operator, that is, the client can be updated based on the stored updated data object during the log compression phase. At the same time, after the update is completed, the second mark bit can also be adjusted to the second mark, such as adjusting isReady to false. In addition, after the update is completed, the first mark bit can also be adjusted to the second mark, such as adjusting hasDDL to false.

[0105] Therefore, when the data structure at the data source end changes, the above-mentioned timing control processing method is used, that is, the write client is first updated based on the coordination operator, and then the write client is subsequently updated based on other operators. The write client update of all operators is completed, so that the data structure information in the avro file and parquet file written to the HDFS system is the latest, updated data structure information. Therefore, there is no need to manually stop the data ingestion process and then restart it, which improves data synchronization efficiency and reduces operation and maintenance costs and difficulties.

[0106] In actual applications, the data source end, deserialization conversion operator, and format conversion operator of the CDC stream scenario are single parallelism, which means that only one processing unit is used to perform operations in parallel during task processing. The data index operator is generally multi-parallelism, which can be represented by partition parallel processing bucket_parallelism. Each parallelism runs in different threads or processes, and is updated and written to the client only when a data structure change event is received. When the format conversion operator forwards data to the data index operator, the forwarding rule is to determine which parallelism in the downstream operator to assign the data to for processing after hashing the key value recordKey of the data. In order to ensure that the coordination operator event constructed based on the data change information in the data structure change event can be sent to each parallelism in the data index operator, the key value of the data structure change event can be constructed to determine the number that can cover the parallelism of the downstream operator as the key value. Therefore, according to the pre-calculation method, the numbers corresponding to all the parallelisms in each operator can be calculated, and the calculated numbers can be used as key values. This can ensure that the coordination operator events broadcast to the downstream operators can be evenly distributed to each parallelism.

[0107] The data lake includes operators for parallel processing, that is, the data lake includes operators with multiple degrees of parallelism, such as data indexing operators, stream data writing operators, and log compression operators.

[0108] In a possible implementation, taking the data index operator as an example, the key value in the data structure change event can be determined based on the target parallelism of the operator in parallel processing in the data lake. The target parallelism target_parallelism is used to indicate the number of units in the parallel processing operator. For example, the default parallelism of the data index operator is 8, that is, the number of units in the data index operator used to parallelly process the data index task is 8. The key value recordKey in the data structure change event refers to a value that can uniquely identify the data change information.

[0109] As an example, you can call the computeDefaultMaxParallelism method of the key value assignment class keyGroupRangeAssignment in the Flink runtime environment Flink Runtime to calculate the maximum parallelism max_parallelism of the operators for parallel processing based on the target parallelism. The maximum parallelism is used to provide a reasonable default parallelism for the key value determination process. The key value assignment class is used to calculate key values ​​that can be evenly distributed to each parallelism, and the computeDefaultMaxParallelism method is used to calculate the maximum parallelism in the process of determining the key value.

[0110] Then, the maximum random value max_randomkey of the operators processed in parallel can be calculated according to the target parallelism. For example, the maximum random value of the operators processed in parallel can be determined by multiplying the target parallelism by 12. The maximum random value is used to indicate the traversal cutoff value of the calculated random number. In the process of determining the key value, some numbers can be exhaustively enumerated as key values, and the parallelism of the downstream operators will be assigned to these numbers when they are used as key values. The number that can eventually cover all parallelisms is used as the final key value.

[0111] After the maximum random value and the maximum parallelism are determined, calculations can be performed based on the maximum parallelism, the maximum random value, and the target parallelism to determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake.

[0112] As an example, a parallelism number allocation processing method can be used to calculate based on the maximum parallelism, the maximum random value and the target parallelism to determine the parallelism numbers of the operators processed in parallel in the data lake and the key values ​​corresponding to them. The parallelism number allocation processing method is used to allocate parallelism numbers to key values.

[0113] For example, you can traverse from 0 to the maximum random value, use each number between 0 and the maximum random value as the key value for traversal calculation, and use the maximum parallelism and target parallelism as parameters to cyclically call the assignKeyToParallelOperator method in the Flink runtime environment Flink Runtime to obtain the parallelism number corresponding to the current number as the key value. If the parallelism number corresponds to other numbers, continue to traverse according to other numbers until the traversal ends. Thus, through the above traversal process, you can get the numbers corresponding to each parallelism number of the corresponding downstream operator, which are the key values ​​corresponding to each parallelism number, thereby ensuring that the coordination operator events broadcast to the downstream operator can be evenly distributed to each parallelism.

[0114] Please refer to Figure 5 , Figure 5 A structural diagram of pre-calculating parallelism provided in an embodiment of the present application, after determining the key value that can cover each parallelism of the downstream operator, a coordination operator event can be constructed based on the key value and data change information. For example, a coordination operator event can be constructed based on the key value and data change information through a constructor, thereby ensuring that the coordination operator event is sent down to each parallelism in the operator.

[0115] As an example, the key value and data change information corresponding to each parallelism number determined above can be traversed to construct a coordination operator event. Then, when forwarding the coordination operator event, the key value can be hashed to determine the parallelism allocation result of the coordination operator event. The parallelism allocation result of the coordination operator event is used to indicate to which parallelism of the downstream operator the coordination operator event constructed based on the current key value is allocated.

[0116] When the parallelism allocation result of the coordination operator event is determined, the coordination operator event can be sent to the parallelism corresponding to the operator for processing according to the parallelism allocation result. Therefore, the above method can ensure that the coordination operator event can be evenly distributed to each parallelism in each downstream operator. It solves the problem of how to accurately synchronize data structure change events to all parallelisms of each downstream operator in a distributed multi-parallelism scenario, improves the accuracy of data synchronization, and can accurately update the write client through the operator in the data lake based on the updated data object.

[0117] The above is a data synchronization method provided by an embodiment of the present application, which includes: obtaining a data structure change event including data change information, that is, first obtaining a data structure change event in which a data structure change occurs. Then, a coordination operator event is constructed based on the data change information, and the coordination operator event is sent to the operator in the data lake, so that the operator in the data lake obtains and stores the updated data object constructed based on the data change information from the coordination operator event. After that, the first mark bit of the operator in the data lake is set to the first mark to mark that the operator in the data lake has stored the updated data object constructed based on the data change information, so that the operator in the data lake can be updated according to the stored updated data object containing the data change information. When entering the cycle of saving the intermediate state, the second mark bit of the operator in the data lake is set to the first mark based on the first mark to mark that the operator in the data lake has met the update conditions and waits for the update opportunity. And based on the updated data object, the client can be updated and written through the coordination operator. Since the coordination operator is used to enable a new processing process, when the processing process enters the cycle of saving the intermediate state, the client can be updated based on the coordination operator to ensure the accuracy of the update timing. Finally, when entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, the write client can be updated based on the updated data object through the operator other than the coordination operator in the data lake. It can be seen that when the processing process enters the next cycle of saving the intermediate state, that is, after completing the update of the coordination operator to write to the client, the update of other operators to write to the client is completed based on the updated data object. The above method can ensure the timing accuracy of updating the data object when the data structure changes, thereby completing the real-time update of the data structure change without stopping and restarting, thereby improving the efficiency of data synchronization.

[0118] The above are some specific implementations of the data synchronization method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.

[0119] See also Figure 6 , which is a schematic diagram of the structure of a data synchronization device 600 provided in an embodiment of the present application, and the device 600 may include:

[0120] The acquisition module 601 is used to acquire a data structure change event; the data structure change event includes data change information;

[0121] A construction module 602, configured to construct a coordination operator event based on the data change information, and send the coordination operator event to operators in the data lake, so that the operators in the data lake obtain and store updated data objects constructed based on the data change information from the coordination operator event;

[0122] A setting module 603, configured to set a first mark bit of the operator in the data lake to a first mark, and when entering a cycle of saving an intermediate state, set a second mark bit of the operator in the data lake to the first mark based on the first mark;

[0123] A first update module 604 is used to update and write to the client through a coordination operator based on the update data object when entering the cycle of saving the intermediate state;

[0124] The second update module 605 is used to update the write client through the operator other than the coordination operator based on the updated data object when entering the next cycle of saving the intermediate state and when the second mark bit of the operator other than the coordination operator in the data lake is the first mark.

[0125] Optionally, the data lake includes operators for parallel processing, and the device further includes:

[0126] A determination unit, configured to determine a key value in the data structure change event based on a target parallelism of an operator processed in parallel in the data lake; the target parallelism is used to indicate the number of units processed in parallel in the operator processed in parallel;

[0127] Correspondingly, the construction module 602 includes:

[0128] The first construction unit is used to construct a coordination operator event based on the key value and the data change information, and send the coordination operator event to the operator in the data lake.

[0129] Optionally, the determining unit includes:

[0130] A first calculation unit, configured to calculate a maximum degree of parallelism of the operators in parallel processing according to a target degree of parallelism of the operators in parallel processing;

[0131] A second calculation unit, used for calculating the maximum random value of the operator processed in parallel according to the target parallelism;

[0132] The third calculation unit is used to perform calculation according to the maximum parallelism, the maximum random value and the target parallelism, and determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake.

[0133] Optionally, the third computing unit includes:

[0134] The fourth computing unit is configured to determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake by performing calculations based on the maximum parallelism, the maximum random value, and the target parallelism by assigning a parallelism number processing method.

[0135] Optionally, the first construction unit includes:

[0136] A second construction unit, configured to construct a coordination operator event based on the key value and the data change information;

[0137] A hash calculation unit, used to perform hash calculation on the key value to determine the parallelism allocation result of the coordination operator event;

[0138] The sending unit is used to send the coordination operator event to the operator in the data lake according to the parallelism allocation result.

[0139] Optionally, the device further comprises:

[0140] An adjustment unit is used to adjust the second mark bit of the operator other than the coordination operator in the data lake to the second mark.

[0141] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0142] The device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program, so that the device executes the data synchronization method described in any embodiment of the present application.

[0143] The computer storage medium stores a computer program. When the code is executed, a device running the computer program implements the data synchronization method described in any embodiment of the present application.

[0144] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0145] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0146] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0147] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtaining a data structure change event; the data structure change event includes data change information; Constructing a coordination operator event based on the data change information, and sending the coordination operator event to the operators in the data lake, so that the operators in the data lake obtain and store the updated data objects constructed based on the data change information from the coordination operator event; Setting a first mark bit of the operator in the data lake to a first mark, and when entering a cycle of saving an intermediate state, setting a second mark bit of the operator in the data lake to the first mark based on the first mark; When entering the cycle of saving the intermediate state, based on the updated data object, the update is written to the client through the coordination operator; When entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, based on the updated data object, the write client is updated by the operator other than the coordination operator.

2. The method according to claim 1, characterized in that: The data lake includes operators for parallel processing, and the acquisition of data structure change events further includes: Determining a key value in the data structure change event based on a target parallelism of an operator processed in parallel in the data lake; the target parallelism is used to indicate the number of units processed in parallel in the operator processed in parallel; The constructing a coordination operator event based on the data change information and sending the coordination operator event to an operator in the data lake includes: A coordination operator event is constructed based on the key value and the data change information, and the coordination operator event is sent to the operator in the data lake.

3. The method according to claim 2, characterized in that The determining the key value in the data structure change event based on the target parallelism of the operators processed in parallel in the data lake includes: Calculating the maximum parallelism of the operators in parallel processing according to the target parallelism of the operators in parallel processing; Calculating the maximum random value of the operators processed in parallel according to the target parallelism; Calculation is performed according to the maximum parallelism, the maximum random value, and the target parallelism to determine key values ​​corresponding to the parallelism numbers of operators processed in parallel in the data lake.

4. The method according to claim 3, characterized in that The calculating according to the maximum parallelism, the maximum random value and the target parallelism to determine the key values ​​corresponding to the parallelism numbers of the operators processed in parallel in the data lake, respectively, includes: By allocating a parallel degree number processing method, calculation is performed based on the maximum parallel degree, the maximum random value and the target parallel degree, and key values ​​corresponding to the parallel degree numbers of operators processed in parallel in the data lake are determined.

5. The method according to claim 2, characterized in that: The constructing a coordination operator event based on the key value and the data change information, and sending the coordination operator event to an operator in the data lake, includes: constructing a coordination operator event based on the key value and the data change information; Performing hash calculation on the key value to determine the parallelism allocation result of the coordination operator event; According to the parallelism allocation result, the coordination operator event is sent to the operator in the data lake.

6. The method according to claim 1, characterized in that When entering the next cycle of saving the intermediate state, when the second mark bit of the operator other than the coordination operator in the data lake is the first mark, based on the updated data object, the write client is updated by the operator other than the coordination operator, and then further includes: The second mark bit of the operators in the data lake except the coordination operator is adjusted to the second mark.

7. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire a data structure change event; the data structure change event includes data change information; A construction module, configured to construct a coordination operator event based on the data change information, and send the coordination operator event to the operator in the data lake, so that the operator in the data lake obtains and stores the updated data object constructed based on the data change information from the coordination operator event; A setting module, configured to set a first mark bit of the operator in the data lake to a first mark, and when entering a cycle of saving an intermediate state, set a second mark bit of the operator in the data lake to the first mark based on the first mark; A first update module, configured to update and write to the client through a coordination operator based on the update data object when entering a cycle of saving the intermediate state; The second update module is used to update the write client through the operator other than the coordination operator based on the updated data object when entering the next cycle of saving the intermediate state and when the second mark bit of the operator other than the coordination operator in the data lake is the first mark.

8. The device according to claim 7, characterized in that The data lake includes operators for parallel processing, and the device further includes: A determination unit, configured to determine a key value in the data structure change event based on a target parallelism of an operator processed in parallel in the data lake; the target parallelism is used to indicate the number of units processed in parallel in the operator processed in parallel; Correspondingly, the construction module includes: The first construction unit is used to construct a coordination operator event based on the key value and the data change information, and send the coordination operator event to the operator in the data lake.

9. A data synchronization device, characterized in that: The device comprises: Memory for storing computer programs; A processor, configured to execute the computer program so that the device executes the data synchronization method as claimed in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the data synchronization method according to any one of claims 1 to 6 is implemented.

Images