A data synchronization method and apparatus

Abstract

The application discloses a data synchronization method and device, in which a data structure change event is acquired; a coordination operator event is constructed based on data change information, and the coordination operator event is issued to an operator in a data lake, so that the operator in the data lake acquires and stores an update data object constructed based on the data change information from the coordination operator event; a first flag bit of the operator in the data lake is set as a first flag, when entering a period of saving an intermediate state, a second flag bit of the operator in the data lake is set as the first flag based on the first flag; when entering the period of saving the intermediate state, a client is updated and written through a coordination operator based on the update data object; when entering a next period of saving the intermediate state, when the second flag bit of the operator other than the coordination operator in the data lake is the first flag, the client is updated and written through the operator other than the coordination operator based on the update data object, and the data synchronization efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data synchronization method and device. BACKGROUND

[0002] With the continuous growth of data, data analysis tasks also increase continuously, and a transactional database at a data source end is good at processing transactional transactions, but is not good at processing correlation analysis between massive data. Therefore, data in the transactional database at the data source end is usually synchronized to a data lake with massive data analysis capability.

[0003] At present, the data lake basically has massive data analysis capability, but as the real-time requirement of the analysis end for data in the data lake is higher and higher, real-time data lake technology emerges as the times require. Through the real-time data lake, business data at the data source end can be stored, that is, the business data at the data source end can be synchronized to the data lake in real time.

[0004] However, if the business data has a structure change, it is necessary to stop the currently running processing process, restart the framework, initialize the write client when restarting, and make the write client update the structure change event. That is, the real-time data lake framework cannot dynamically process the structure of data and change events, and needs to stop the data reading process by manual operation, and restart, thereby reducing the data synchronization efficiency. SUMMARY

[0005] Therefore, the embodiments of the present application provide a data synchronization method and device, aiming to improve the data synchronization efficiency.

[0006] In a first aspect, the embodiments of the present application provide a data synchronization method, which comprises:

[0007] obtaining a data structure change event; the data structure change event comprises data change information;

[0008] constructing a coordination operator event based on the data change information, and issuing the coordination operator event to an operator in the data lake, so that the operator in the data lake obtains and stores an updated data object constructed based on the data change information from the coordination operator event;

[0009] setting a first flag bit of the operator in the data lake to a first flag, and setting a second flag bit of the operator in the data lake to the first flag based on the first flag when entering a period of saving an intermediate state;

[0010] updating the write client through the coordination operator based on the updated data object when entering the period of saving the intermediate state;

[0011] When a second flag bit of an operator other than the coordination operator in the data lake is the first flag bit when entering a cycle of saving an intermediate state, based on the update data object, the write client is updated by the operator other than the coordination operator.

[0012] Optionally, the data lake includes an operator processed in parallel, the obtaining a data structure change event further includes:

[0013] Based on a target parallelism of the operator processed in parallel in the data lake, a key value in the data structure change event is determined; the target parallelism is used to indicate a number of units processed in parallel in the operator processed in parallel;

[0014] The constructing a coordination operator event based on the data change information and the dispatching the coordination operator event to the operator in the data lake includes:

[0015] The constructing a coordination operator event based on the key value and the data change information and the dispatching the coordination operator event to the operator in the data lake.

[0016] Optionally, the determining the key value in the data structure change event based on the parallelism number of the operator processed in parallel in the data lake includes:

[0017] According to the target parallelism of the operator processed in parallel, a maximum parallelism of the operator processed in parallel is calculated;

[0018] According to the target parallelism, a maximum random value of the operator processed in parallel is calculated;

[0019] According to the calculation based on the maximum parallelism, the maximum random value and the target parallelism, a key value corresponding to the parallelism number of the operator processed in parallel in the data lake is determined.

[0020] Optionally, the determining the key value corresponding to the parallelism number of the operator processed in parallel in the data lake based on the calculation based on the maximum parallelism, the maximum random value and the target parallelism includes:

[0021] According to the calculation based on the maximum parallelism, the maximum random value and the target parallelism, a key value corresponding to the parallelism number of the operator processed in parallel in the data lake is determined by a parallelism number processing method.

[0022] Optionally, the constructing a coordination operator event based on the key value and the data change information and the dispatching the coordination operator event to the operator in the data lake includes:

[0023] construct a coordination operator event based on the key value and the data change information;

[0024] perform hash calculation on the key value to determine parallelism allocation result of the coordination operator event;

[0025] according to the parallelism allocation result, issue the coordination operator event to an operator in a data lake.

[0026] Optionally, when a second flag bit of the operator in the data lake other than the coordination operator is the first flag when entering a period of saving an intermediate state, the write client is updated by the operator other than the coordination operator based on the update data object, and then the method further comprises:

[0027] adjust the second flag bit of the operator in the data lake other than the coordination operator to the second flag.

[0028] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0029] an obtaining module configured to obtain a data structure change event; the data structure change event comprising data change information;

[0030] a constructing module configured to construct a coordination operator event based on the data change information, and issue the coordination operator event to an operator in a data lake, so that the operator in the data lake obtains and stores an update data object constructed based on the data change information from the coordination operator event;

[0031] a setting module configured to set a first flag bit of the operator in the data lake to a first flag, and set a second flag bit of the operator in the data lake to the first flag based on the first flag when entering a period of saving an intermediate state;

[0032] a first updating module configured to update a write client by a coordination operator based on the update data object when entering a period of saving an intermediate state;

[0033] a second updating module configured to update the write client by an operator other than the coordination operator based on the update data object when a second flag bit of the operator in the data lake other than the coordination operator is the first flag when entering a period of saving an intermediate state.

[0034] Optionally, the data lake comprises an operator for parallel processing, and the device further comprises:

[0035] A determining unit is configured to determine a key value in the data structure change event based on a target parallelism of an operator in parallel processing in the data lake, where the target parallelism is used to indicate a number of units in parallel processing in the operator in parallel processing.

[0036] Correspondingly, the construction module comprises:

[0037] A first construction unit is configured to construct a coordination operator event based on the key value and the data change information, and to issue the coordination operator event to an operator in the data lake.

[0038] Optionally, the determining unit comprises:

[0039] A first calculation unit is configured to calculate a maximum parallelism of the operator in parallel processing according to the target parallelism of the operator in parallel processing.

[0040] A second calculation unit is configured to calculate a maximum random value of the operator in parallel processing according to the target parallelism.

[0041] A third calculation unit is configured to determine a key value corresponding to a parallelism number of the operator in parallel processing in the data lake based on the maximum parallelism, the maximum random value and the target parallelism.

[0042] Optionally, the third calculation unit comprises:

[0043] A fourth calculation unit is configured to determine a key value corresponding to a parallelism number of the operator in parallel processing in the data lake based on the maximum parallelism, the maximum random value and the target parallelism by using a parallelism number processing method.

[0044] Optionally, the first construction unit comprises:

[0045] A second construction unit is configured to construct a coordination operator event based on the key value and the data change information.

[0046] A hash calculation unit is configured to perform hash calculation on the key value to determine a parallelism allocation result of the coordination operator event.

[0047] An issuing unit is configured to issue the coordination operator event to an operator in the data lake according to the parallelism allocation result.

[0048] Optionally, the apparatus further comprises:

[0049] An adjusting unit is configured to adjust a second flag bit of an operator other than the coordination operator in the data lake to a second flag.

[0050] In a third aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0051] a memory for storing a computer program;

[0052] a processor for executing the computer program to cause the device to perform the data synchronization method of the first aspect.

[0053] In a fourth aspect, an embodiment of the present application provides a computer storage medium, the computer readable storage medium storing a computer program, when the computer program is run, a device running the computer program implements the data synchronization method of the first aspect.

[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0055] The embodiments of the present application provide a data synchronization method and device, the method comprising: obtaining a data structure change event comprising data change information, that is, first obtaining a data structure change event in which data structure change occurs. Then, constructing a reconciliation operator event based on the data change information, and issuing the reconciliation operator event to the operators in the data lake, so that the operators in the data lake obtain and store an updated data object constructed based on the data change information from the reconciliation operator event. Then, setting a first flag bit of the operators in the data lake to a first flag to mark that the operators in the data lake have stored the updated data object constructed based on the data change information, so that the operators in the data lake can update according to the stored updated data object containing the data change information. When entering a period of saving an intermediate state, setting a second flag bit of the operators in the data lake to the first flag based on the first flag, to mark that the operators in the data lake have the update condition and wait for the update opportunity. And the write client can be updated by the reconciliation operator based on the updated data object. Since the reconciliation operator is used to enable a new processing process, when the processing process enters the period of saving the intermediate state, the write client can be updated based on the reconciliation operator first to ensure the accuracy of the update timing. Finally, when entering the next period of saving the intermediate state, when the second flag bit of the operators other than the reconciliation operator in the data lake is the first flag, the write client can be updated by the operators other than the reconciliation operator in the data lake based on the updated data object. It can be seen that when the processing process enters the next period of saving the intermediate state, that is, after the write client is updated by the reconciliation operator, the write client is updated by other operators based on the updated data object. Through the above method, the timing accuracy of the updated data object can be ensured when the data structure changes, so that real-time updating of the data structure change is completed without stopping and restarting, and the data synchronization efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to make the technical solution of the present embodiment or the prior art clearer, the drawings needed in the embodiment or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 The application scenario of the data synchronization method provided by the embodiment of the present application is shown in the figure.

[0058] Figure 2 The flowchart of the data synchronization method provided by the embodiment of the present application is shown in the figure.

[0059] Figure 3 The structure diagram of data processing provided by the embodiment of the present application is shown in the figure.

[0060] Figure 4 The timing diagram of data synchronization provided by the embodiment of the present application is shown in the figure.

[0061] Figure 5 The structure diagram of pre-computed parallelism provided by the embodiment of the present application is shown in the figure.

[0062] Figure 6 The structure diagram of the data synchronization device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0063] In order to make the technical solution of the present embodiment or the prior art clearer, the drawings needed in the embodiment or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Change data capture (CDC) is a real-time extract-transform-load (ETL) method, mainly used in the fields of data integration and analysis in enterprise informatization planning and implementation process.

[0065] With the continuous growth of data, data analysis tasks also increase continuously, and the transactional database at the data source end is good at processing transactional transactions, but not good at processing correlation analysis between massive data. Therefore, the data in the transactional database at the data source end is usually synchronized to the data lake with massive data analysis capability. The data lake refers to a system for storing various raw data, and has the ability of storage, calculation and query.

[0066] Traditional data lake is based on open source distributed file system (Hadoop Distributed File System, HDFS) as a storage base, and data warehouse Hive as a computing query engine, so that the data lake basically has massive data analysis capability. However, as the real-time requirement of the analysis end for the data in the data lake is getting higher and higher, such as real-time admission and discharge statistics, ward treatment, and real-time operation in the medical field, real-time data lake technology emerges as the times require. Through real-time data lake, business data from the data source end can be stored, that is, business data from the data source end can be synchronized to the data lake in real time.

[0067] At present, data lake engine Hudi is usually used as a real-time data lake engine scheme in various industries. Hudi has good update, deletion and other performance and rich query capabilities, and has high community activity. Flink is a stream computing engine. In the scheme of real-time data lake of CDC data, Flink+Hudi stream data import framework is usually formed with Hudi framework, which has been widely used in many fields, such as e-commerce, logistics and finance.

[0068] In the process of real-time data lake, in addition to synchronizing the data from the data source end to the data lake in real time, the data modify language (DML) of the data is also synchronized to the data lake in real time. The data under the Hudi framework will generate avro file and parquet file of big data file storage format, which includes data content. In addition, if the data structure in the data source end changes, such as storing data in table structure, the table structure changes, such as adding or reducing a column of data, the data structure change should also be updated to the data lake. Under the Flink+Hudi native framework, only the data structure such as table structure is synchronized at the beginning of initialization. In the running process, the data definition language (DDL) usually refers to the table structure change, so that the table structure in the data lake cannot keep consistent with the data source end (i.e. business source end), thereby affecting the data analysis in the data lake.

[0069] However, if the business data has a structural change, the currently running processing process needs to be stopped, i.e., the processing process running under the Flink+Hudi framework is stopped, the framework is restarted, and the write client is initialized when the framework is restarted, so that the write client updates the structural change event, i.e., the changed data structure is included in the written avro file and parquet file. That is, the real-time data lake framework cannot dynamically process the structure of the data and the change event, and needs to stop the data reading process by manual operation, and restart, which reduces the data synchronization efficiency.

[0070] Therefore, to solve the above problems, in the embodiments of the present application, a data structure change event including data change information is obtained, i.e., a data structure change event in which data structure change occurs is obtained first. Then, a coordination operator event is constructed based on the data change information, and the coordination operator event is issued to the operators in the data lake, so that the operators in the data lake obtain and store the updated data object constructed based on the data change information from the coordination operator event. After that, the first flag bit of the operator in the data lake is set to the first flag to mark that the operator in the data lake has stored the updated data object constructed based on the data change information, so that the operator in the data lake can update according to the stored updated data object containing the data change information. When entering the period of saving the intermediate state, the second flag bit of the operator in the data lake is set to the first flag based on the first flag, to mark that the operator in the data lake has the update condition and waits for the update opportunity. And the write client can be updated by the coordination operator based on the updated data object. Since the coordination operator is used to enable a new processing process, when the processing process enters the period of saving the intermediate state, the write client can be updated based on the coordination operator first to ensure the accuracy of the update timing. Finally, when entering the next period of saving the intermediate state, when the second flag bit of the operator in the data lake except the coordination operator is the first flag, the write client can be updated by the operator in the data lake except the coordination operator based on the updated data object. It can be seen that when the processing process enters the next period of saving the intermediate state, i.e., after the write client is updated by the coordination operator, the write client is updated by other operators based on the updated data object. Through the above method, the timing accuracy of the updated data object can be ensured when the data structure changes, so that the real-time update of the data structure change is completed without stopping and restarting, and the data synchronization efficiency is improved.

[0071] For example, one of the scenarios of the embodiments of the present application can be applied to a data lake framework such as Figure 1The scenario shown. The scenario includes a database 101 and a server 102, wherein the database 101 includes a data structure change event, and the server 102 obtains the data structure change event including data change information from the database 101 by using the implementation provided by the embodiment of the present application, to perform data synchronization based on the data structure change event.

[0072] First, in the above application scenario, although the actions of the implementation provided by the embodiment of the present application are executed by the server 102; however, the embodiment of the present application is not limited in terms of execution subject, as long as the actions disclosed by the implementation provided by the embodiment of the present application are executed.

[0073] Secondly, the above scenario is only one scenario example provided by the embodiment of the present application, and the embodiment of the present application is not limited to this scenario.

[0074] The specific implementation of the data synchronization method and device in the embodiment of the present application will be described in detail below with reference to the accompanying drawings and by embodiments.

[0075] Referring to Figure 2 , the figure is a flowchart of a data synchronization method provided by the embodiment of the present application, in combination with Figure 2 , specifically can include:

[0076] S201: Obtain a data structure change event, and the data structure change event includes data change information.

[0077] The data change information refers to the data information changed in the data source end, for example, the data information in the newly added column in the changed table structure in the data providing business source end, or the metadata information in the changed table structure, such as the newly added row of data in the table structure, etc. The data change information is not specifically limited by the present application. The data structure change event refers to an event that the data structure is changed, such as the data in the data source end is stored in the form of a table structure, a column of data is added or a column of data is reduced in the table structure, etc. The data structure change event includes data change information, such as the data information in the changed table structure.

[0078] S202: Construct a coordination operator event based on the data change information, and issue the coordination operator event to the operator in the data lake, so that the operator in the data lake obtains and stores an update data object constructed based on the data change information from the coordination operator event.

[0079] The overall data processing sequence in the Flink-Hudi framework is as shown in Figure 3 , Figure 3The structural diagram of data processing provided by the embodiment of the present application first performs source-side log mining on the data source end Source, mines the required processing data (i.e., data change information that has changed) from the business log of the data source end, then performs deserialization conversion on the data through a deserialization conversion operator, and performs format conversion on the deserialization-converted data through a format conversion operator to convert the data into a data format that can be streamed in the Flink framework. Subsequently, the data indexing stage is entered, the data indexing operator is used to locate the corresponding storage location in the HDFS system according to the key value Key of the data, the data change information is written into the HDFS file through the stream data writing operator, and the coordinator can be notified by the coordination operator to synchronize the data to the Hive after the data writing is completed. At the same time, the event notification can be sent to the downstream operator to generate a compression merging plan, and the log compression is performed through the log compression operator to complete the result submission.

[0080] Among the above-mentioned various processing processes, the processing order needs to be executed to ensure the accuracy of the timing, and the timing function and sequence cohesion are high. The deserialization conversion operator is used to convert the serialized data (such as byte sequence or string form) back to its original object or data structure; the format conversion operator is used to convert the deserialization-converted data into a data format that can be streamed in the Flink framework; the data indexing operator is used to locate the storage location of the data in the HDFS system; the stream data writing operator is used to write the data change information into the HDFS file; the coordination operator is used to start a new processing process timeline server, and is used to synchronize the updated data object to the Hive; and the log compression operator is used to reduce the storage space occupied by the log.

[0081] After the deserialization operator obtains the data structure change event, a new complete updated data object newConfig can be constructed based on the data change information in the data structure change event, and then a coordination operator event CEvent is constructed, the coordination operator event includes the updated data object, and then the coordination operator event can be sent to the downstream operator, so that the downstream operator in the data lake can obtain and store the updated data object constructed based on the data change information from the coordination operator event.

[0082] The updated data object refers to an object that is reconstructed based on the data change information and is suitable for updating the data structure in the data lake, and the coordination operator event refers to an event used to notify the coordinator. The coordinator refers to a device in the data lake that receives query requests from the client, decomposes these requests into a series of tasks, and then assigns the tasks to the operators in the data lake for processing.

[0083] S203: set the first mark bit of the operator in the data lake to the first mark, and set the second mark bit of the operator in the data lake to the first mark based on the first mark when entering the period of saving the intermediate state.

[0084] The first mark bit of the operator refers to the mark bit inside the operator in the data lake, which is used to indicate whether the operator has received the coordination operator event to take out the updated data object from the coordination operator event; the first mark is used to indicate that the operator has received the coordination operator event, i.e., the data structure change has occurred. For example, the first mark bit of the operator can be represented by whether the data structure change has occurred hasDDL, and the first mark can be represented by true. When hasDDL is true, it means that the operator has received the coordination operator event and can take out the updated data object from the coordination operator event. Conversely, when hasDDL is false, it means that the operator has not received the coordination operator event and cannot take out the updated data object from the coordination operator event, i.e., no data structure change has occurred. In addition, hasDDL is false by default.

[0085] As an example, reference can be made to Figure 4 , Figure 4 The timing diagram of data synchronization provided by the embodiments of the present application. When the data index operator receives the coordination operator event, the internal mark bit hasDDL of the data index operator can be set to true, so that the data index operator can take out the updated data object newConfig from the coordination operator event and save it in the local attribute, while continuing to send the coordination operator event downstream;

[0086] When the stream data write operator receives the coordination operator event, the internal mark bit hasDDL of the stream data write operator can be set to true, so that the stream data write operator can take out the updated data object newConfig from the coordination operator event and save it in the local attribute, while continuing to send the coordination operator event downstream;

[0087] When the coordination operator receives the coordination operator event, the internal mark bit hasDDL of the coordination operator can be set to true.

[0088] When the log compression operator receives the coordination operator event, the internal mark bit hasDDL of the log compression operator can be set to true, so that the log compression operator can take out the updated data object newConfig from the coordination operator event and save it in the local attribute.

[0089] After the above-mentioned response processing steps of different operators to the coordination operator event, the respective mark bits and attribute information are updated, so that each operator stores the updated data object.

[0090] Then when the first flag bit of the operator in the data lake is the first flag when entering the checkpoint of saving the intermediate state, the second flag of the operator can be set to the first flag by the device.

[0091] Checkpoint in the Flink-Hudi framework is used to save the intermediate state of the job during stream processing. The state information of the Flink task can be periodically saved to the persistent storage medium, so that the execution state of the task can be recovered in time when the task fails, thereby realizing the fault tolerance of the data structure update task.

[0092] The second flag bit of the operator refers to the flag bit of the operator content in the data lake, which is used to indicate whether the coordination operator has completed the update, that is, when entering the checkpoint of saving the intermediate state, the coordination operator will complete the update based on the update data object in advance, and other operators will complete the update after the coordination operator updates. Since the coordination operator is used to start a new processing process in the data lake, other operators in the data lake access this processing process, and only when the coordination operator is updated can the time sequence accuracy of data update be ensured. When the second flag bit is the first flag, it is used to indicate that the operator has the update condition and only needs to wait for the update opportunity, that is, the operator has stored the update data object, and after the coordination operator completes the update, the update can be performed. For example, the second flag bit of the operator can be represented by whether the coordination operator is ready isReady, and the first flag can be represented by true. When isReady is true, it means that the coordination operator has completed the update in this period, and the operator has the update condition and can be updated in the next period. Conversely, when isReady is false, it means that the coordination operator has not completed the update in this period, and the operator has the update condition but does not have the update opportunity. In addition, isReady is false by default.

[0093] As an example, when the hasDDL of the data index operator is true, when entering the checkpoint of saving the intermediate state, such as the i-th checkpoint of the data index operator, the internal flag bit isReady of the operator can be set to true;

[0094] When the hasDDL of the stream data writing operator is true, when entering the checkpoint of saving the intermediate state, such as the i-th checkpoint of the stream data writing operator, the internal flag bit isReady of the operator can be set to true;

[0095] When entering the checkpoint of saving the intermediate state, such as the i-th checkpoint of the coordination operator, the coordination operator updates the writing client.

[0096] In the case that the hasDDL of the log compression operator is true, when entering the period of saving the intermediate state, such as the log compression operator entering the ith checkpoint, the internal flag bit isReady of the operator can be set to true.

[0097] S204: When entering the period of saving the intermediate state, the write client is updated by the reconciliation operator based on the update data object.

[0098] When entering the checkpoint of saving the intermediate state, the reconciliation operator can update the write client based on the update data object, so that the write client can be updated based on the update data object in the reconciliation operator stage.

[0099] The write client is used to indicate the program processing object responsible for writing data, that is, the client responsible for writing the update data object into the avro file and the parquet file.

[0100] S205: When entering the next period of saving the intermediate state, when the second flag bit of the operator other than the reconciliation operator in the data lake is the first flag, the write client is updated by the operator other than the reconciliation operator based on the update data object.

[0101] When entering the next period of saving the intermediate state, the second flag bit of the operator other than the reconciliation operator in the data lake can be judged, and if the second flag bit of the operator other than the reconciliation operator in the data lake is the first flag, it indicates that the other operator in the data lake can update the write client according to the stored update data object.

[0102] As an example, when the data index operator enters the next period of saving the intermediate state, such as the data index operator entering the ith+1 checkpoint, it is judged whether isReady is true. If true, the write client can be updated by the data index operator based on the update data object stored by the operator, that is, the write client can be updated based on the stored update data object in the data index stage. At the same time, after completing the update, the second flag bit can also be adjusted to the second flag, such as adjusting isReady to false, and the second flag is used to indicate that the update is not completed, so as to facilitate subsequent continuous data synchronization update based on new data change information, which helps to improve the data synchronization efficiency. In addition, after completing the update, the first flag bit can also be adjusted to the second flag, such as adjusting hasDDL to false.

[0103] When the stream data write operator enters the next period of saving the intermediate state, such as the stream data write operator entering the i+1 checkpoint, it is determined whether isReady is true. If true, the write client can be updated by the stream data write operator based on the update data object stored by the operator, that is, the write client can be updated based on the stored update data object in the stream data write phase. At the same time, after the update is completed, the second flag bit can be adjusted to the second mark, such as adjusting isReady to false. In addition, after the update is completed, the first flag bit can be adjusted to the second mark, such as adjusting hasDDL to false.

[0104] When the log compression operator enters the next period of saving the intermediate state, such as the log compression operator entering the i+1 checkpoint, it is determined whether isReady is true. If true, the write client can be updated by the log compression operator based on the update data object stored by the operator, that is, the write client can be updated based on the stored update data object in the log compression phase. At the same time, after the update is completed, the second flag bit can be adjusted to the second mark, such as adjusting isReady to false. In addition, after the update is completed, the first flag bit can be adjusted to the second mark, such as adjusting hasDDL to false.

[0105] Thus, when the data structure of the data source end changes, the write client of all operators is updated by the above timing control processing method, that is, the write client is first updated based on the coordination operator, and then the write client is updated based on other operators, so that the data structure information in the avro file and the parquet file written to the HDFS system is the latest, and the updated data structure information, so that manual operation is not required to stop the data ingestion process and restart, thereby improving the data synchronization efficiency and reducing the operation and maintenance cost and difficulty.

[0106] In practical applications, the data source end, the deserialization conversion operator and the format conversion operator of the CDC stream scenario are single parallelism, which means that only one processing unit is used to perform parallel operations during task processing. The data index operator is generally multi-parallelism, which can be represented by partition parallel processing bucket_parallelism. Each parallelism runs in a different thread or process, and the write client is only updated in the case of receiving a data structure change event. When the format conversion operator forwards data to the data index operator, the forwarding rule is to determine which parallelism in the downstream operator to process the data by hashing the data key recordKey. In order to ensure that the coordination operator event constructed based on the data change information in the data structure change event can be sent to each parallelism in the data index operator, the key value of the data structure change event can be constructed to determine the number that can cover the parallelism of the downstream operator as the key value. Thus, the number corresponding to each parallelism in each operator can be calculated according to the pre-computation method, and the calculated number is used as the key value, so that the coordination operator event broadcast to the downstream operator can be evenly distributed to each parallelism.

[0107] The operators in the data lake include parallel processing, i.e., the operators in the data lake include multi-parallelism, such as the data index operator, the stream data write operator, and the log compression operator.

[0108] In a possible implementation, taking the data index operator as an example, the key value in the data structure change event can be determined based on the target parallelism of the parallel processing operator in the data lake. The target parallelism target_parallelism is used to indicate the number of parallel processing units in the parallel processing operator, for example, the default parallelism of the data index operator is 8, i.e., the number of units for parallel processing of the data index task in the data index operator is 8. The key value recordKey in the data structure change event refers to a value that can uniquely identify the data change information.

[0109] As an example, the computeDefaultMaxParallelism method of the key value assignment class keyGroupRangeAssignment in the Flink runtime environment Flink Runtime can be called to calculate the maximum parallelism max_parallelism of the parallel processing operator based on the target parallelism. The maximum parallelism is used to provide a reasonable default parallelism for the key value determination process. The key value assignment class is used to calculate the key value that can be evenly distributed to each parallelism, and the computeDefaultMaxParallelism method is used to calculate the maximum parallelism in the key value determination process.

[0110] Then, the maximum random value max_randomkey of the parallel processing operator can be calculated according to the target parallelism, for example, the maximum random value of the parallel processing operator can be determined by multiplying the target parallelism by 12, and the maximum random value is used to indicate the traversal cutoff value of the calculated random number. In the process of determining the key value, some numbers can be exhausted as key values, and it is calculated which parallelism of the downstream operator will be assigned when these numbers are used as key values, and the numbers that can finally cover all parallelisms are used as the final key values.

[0111] After the maximum random value and the maximum parallelism are determined, the key values corresponding to the parallelism numbers of the parallel processing operators in the data lake can be calculated based on the maximum parallelism, the maximum random value and the target parallelism.

[0112] As an example, the key values corresponding to the parallelism numbers of the parallel processing operators in the data lake can be calculated based on the maximum parallelism, the maximum random value and the target parallelism by using the parallelism number assignment method. The parallelism number assignment method is used to assign the parallelism numbers to the key values.

[0113] For example, the traversal can be performed from 0 to the maximum random value, and the traversal calculation is performed for each number between 0 and the maximum random value as a key value, and the assignKeyToParallelOperator method in the Flink Runtime is called in a loop with the maximum parallelism and the target parallelism as parameters, to obtain the parallelism number corresponding to the current number as a key value, if the parallelism number corresponds to other numbers, then the traversal is continued according to the other numbers until the traversal is completed. Thus, through the above traversal process, the numbers corresponding to each parallelism number of the corresponding downstream operator can be obtained, and these numbers are the key values corresponding to each parallelism number, so that the coordination operator event broadcast to the downstream operator can be uniformly distributed to each parallelism.

[0114] Reference can be made to Figure 5 , Figure 5 The structure diagram for pre-calculating parallelism provided by the embodiments of the present application, after determining the key values that can cover each parallelism of the downstream operator, the coordination operator event can be constructed based on the key values and the data change information, for example, the coordination operator event can be constructed based on the key values and the data change information by using a constructor, so that the coordination operator event can be issued to each parallelism in the operator.

[0115] As an example, a coordination operator event can be constructed using the key value and data change information corresponding to each parallelism number determined above. When forwarding the coordination operator event, the key value can be hashed to determine the parallelism allocation result of the coordination operator event. The parallelism allocation result of the coordination operator event indicates which parallelism of the downstream operator the coordination operator event constructed based on the current key value is allocated to.

[0116] After determining the parallelism allocation result of the coordination operator event, the coordination operator event can be issued to the parallelism corresponding to the operator according to the parallelism allocation result for processing. Thus, the above method can ensure that the coordination operator event can be evenly allocated to each parallelism in each operator downstream. The problem of how to accurately synchronize the data structure change event to all parallelisms of each operator downstream in a distributed multi-parallelism scenario is solved, the accuracy of data synchronization is improved, and the write client can be effectively updated based on the updated data object through the operator in the data lake.

[0117] The above is a data synchronization method provided by an embodiment of the present application, which comprises: obtaining a data structure change event comprising data change information, that is, first obtaining a data structure change event in which a data structure change occurs. Then, a reconciliation operator event is constructed based on the data change information, and the reconciliation operator event is issued to an operator in the data lake, so that the operator in the data lake obtains and stores an update data object constructed based on the data change information from the reconciliation operator event. After that, a first flag bit of the operator in the data lake is set to a first flag to mark that the operator in the data lake has stored the update data object constructed based on the data change information, so that the operator in the data lake can be updated according to the stored update data object containing the data change information. When entering a period of saving an intermediate state, a second flag bit of the operator in the data lake is set to the first flag based on the first flag to mark that the operator in the data lake has the update condition and waits for an update opportunity. And the write client can be updated by the reconciliation operator based on the update data object. Since the reconciliation operator is used to enable a new processing process, when the processing process enters the period of saving the intermediate state, the write client can be updated based on the reconciliation operator first to ensure the accuracy of the update timing. Finally, when entering a next period of saving the intermediate state, when the second flag bit of the operator other than the reconciliation operator in the data lake is the first flag, the write client can be updated by the operator other than the reconciliation operator in the data lake based on the update data object. It can be seen that when the processing process enters the next period of saving the intermediate state, that is, after the write client is updated by the reconciliation operator, the write client is updated by other operators based on the update data object. Through the above method, the timing accuracy of the update data object can be ensured when the data structure changes, so that real-time update of the data structure change is completed without stopping and restarting, and the data synchronization efficiency is improved.

[0118] The above is some specific implementation manners of the data synchronization method provided by an embodiment of the present application, based on which, the present application further provides a corresponding device. The device provided by an embodiment of the present application will be introduced from the perspective of functional modularization.

[0119] Referring to Figure 6 The figure is a structural schematic diagram of a data synchronization device 600 provided by an embodiment of the present application, which can comprise:

[0120] The obtaining module 601 is configured to obtain a data structure change event, wherein the data structure change event comprises data change information;

[0121] The constructing module 602 is configured to construct a coordination operator event based on the data change information, and distribute the coordination operator event to the operators in the data lake, so that the operators in the data lake obtain and store an update data object constructed based on the data change information from the coordination operator event.

[0122] The setting module 603 is configured to set a first flag bit of the operators in the data lake to a first flag, and set a second flag bit of the operators in the data lake to the first flag based on the first flag when entering a period of saving an intermediate state.

[0123] The first updating module 604 is configured to update the write client through the coordination operator based on the update data object when entering the period of saving the intermediate state.

[0124] The second updating module 605 is configured to update the write client through the operators other than the coordination operator based on the update data object when entering a next period of saving an intermediate state, and when the second flag bit of the operators other than the coordination operator in the data lake is the first flag.

[0125] Optionally, the data lake comprises the operators processed in parallel, and the apparatus further comprises:

[0126] The determining unit is configured to determine a key value in the data structure change event based on a target parallelism degree of the operators processed in parallel in the data lake; the target parallelism degree is used to indicate a number of units processed in parallel in the operators processed in parallel.

[0127] Correspondingly, the constructing module 602 comprises:

[0128] The first constructing unit is configured to construct a coordination operator event based on the key value and the data change information, and distribute the coordination operator event to the operators in the data lake.

[0129] Optionally, the determining unit comprises:

[0130] The first calculating unit is configured to calculate a maximum parallelism degree of the operators processed in parallel according to the target parallelism degree of the operators processed in parallel.

[0131] The second calculating unit is configured to calculate a maximum random value of the operators processed in parallel according to the target parallelism degree.

[0132] The third calculating unit is configured to calculate the parallelism degree number of the operators processed in parallel in the data lake according to the maximum parallelism degree, the maximum random value and the target parallelism degree, and determine the key value corresponding to the parallelism degree number respectively.

[0133] Optionally, the third computing unit comprises:

[0134] The fourth computing unit is configured to determine the parallel degree number corresponding to the key value in the data lake by distributing the parallel degree number processing method based on the maximum parallel degree, the maximum random value and the target parallel degree.

[0135] Optionally, the first constructing unit comprises:

[0136] The second constructing unit is configured to construct a coordination operator event based on the key value and the data change information.

[0137] The hash computing unit is configured to perform hash calculation on the key value to determine the parallel degree distribution result of the coordination operator event.

[0138] The delivering unit is configured to deliver the coordination operator event to the operator in the data lake according to the parallel degree distribution result.

[0139] Optionally, the apparatus further comprises:

[0140] The adjusting unit is configured to adjust the second mark bit of the operator in the data lake except the coordination operator to a second mark.

[0141] Embodiments of the present application further provide a corresponding device and a computer storage medium for implementing the scheme provided by the embodiments of the present application.

[0142] The device comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to execute the computer program to enable the device to perform the data synchronization method described in any embodiment of the present application.

[0143] The computer storage medium stores a computer program, and when the code is executed, the device running the computer program implements the data synchronization method described in any embodiment of the present application.

[0144] The "first" and "second" in the names mentioned in the embodiments of the present application are only used for name identification, and do not represent the first and second in order.

[0145] Those skilled in the art can clearly understand that all or part of the steps of the above-mentioned method embodiments can be implemented by means of software plus a general hardware platform through the description of the above embodiments. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0146] It should be noted that each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments. The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components prompted as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0147] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed in the present application can be easily thought of by those skilled in the art without creative labor, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data synchronization method, characterized in that, The method includes: Obtain data structure change events; the data structure change events include data change information; Based on the data change information, a coordination operator event is constructed, and the coordination operator event is sent to the operators in the data lake, so that the operators in the data lake can obtain and store the updated data object constructed based on the data change information from the coordination operator event; The first flag bit of the operator in the data lake is set to the first flag. When entering the period of saving intermediate state, the second flag bit of the operator in the data lake is set to the first flag based on the first flag. The first flag bit is a flag bit inside the operator in the data lake, which is used to indicate whether the operator has received a coordination operator event so as to retrieve the updated data object from the coordination operator event. When entering the period of saving intermediate states, the updated data object is updated and written to the client through the coordination operator; When entering the next cycle of saving intermediate states, when the second flag bit of the operator other than the coordination operator in the data lake is the first flag, the write client is updated based on the updated data object by the operator other than the coordination operator; the second flag bit is the flag bit of the operator content in the data lake, used to indicate whether the coordination operator has completed the update.

2. The method according to claim 1, characterized in that, The data lake includes operators for parallel processing. Following the acquisition of data structure change events, the process further includes: Based on the target parallelism of the parallel processing operators in the data lake, the key values ​​in the data structure change event are determined; the target parallelism is used to indicate the number of parallel processing units in the parallel processing operators. The step of constructing a coordination operator event based on the data change information and distributing the coordination operator event to the operators in the data lake includes: Based on the key values ​​and the data change information, a coordination operator event is constructed, and the coordination operator event is sent to the operators in the data lake.

3. The method according to claim 2, characterized in that, The determination of key values ​​in the data structure change event based on the target parallelism of the operators in the data lake includes: Calculate the maximum parallelism of the parallel processing operator based on the target parallelism of the parallel processing operator; Calculate the maximum random value of the parallel processing operator based on the target parallelism. Based on the maximum parallelism, the maximum random value, and the target parallelism, key values ​​corresponding to the parallelism numbers of the operators in the data lake are determined.

4. The method according to claim 3, characterized in that, The step of calculating and determining key values ​​corresponding to the parallelism numbers of the operators in the data lake based on the maximum parallelism, the maximum random value, and the target parallelism includes: By using a parallelism numbering processing method, key values ​​corresponding to the parallelism numbers of the operators in the data lake are determined based on the maximum parallelism, the maximum random value, and the target parallelism.

5. The method according to claim 2, characterized in that, The process of constructing a coordination operator event based on the key value and the data change information, and then distributing the coordination operator event to the operators in the data lake, includes: Construct a coordination operator event based on the key values ​​and the data change information; The key value is hashed to determine the parallelism allocation result of the coordination operator event; Based on the parallelism allocation result, the coordination operator event is sent to the operators in the data lake.

6. The method according to claim 1, characterized in that, When entering the next cycle of saving intermediate states, if the second flag bit of an operator other than the coordination operator in the data lake is the first flag, the write client is updated based on the updated data object using operators other than the coordination operator, and then the process further includes: Adjust the second flag bit of the operators in the data lake other than the coordination operator to the second flag.

7. A data synchronization device, characterized in that, The device includes: The acquisition module is used to acquire data structure change events; the data structure change events include data change information. The construction module is used to construct a coordination operator event based on the data change information and send the coordination operator event to the operators in the data lake, so that the operators in the data lake can obtain and store the updated data object constructed based on the data change information from the coordination operator event; The setting module is used to set the first flag bit of the operator in the data lake to the first flag, and when entering the period of saving intermediate state, it sets the second flag bit of the operator in the data lake to the first flag based on the first flag; the first flag bit is the flag bit inside the operator in the data lake, which is used to indicate whether the operator has received a coordination operator event, so as to retrieve the updated data object from the coordination operator event; The first update module is used to update and write to the client based on the update data object through a coordination operator when entering the period of saving intermediate state; The second update module is used to update the write client based on the updated data object when entering the next period of saving intermediate states, and when the second flag bit of the operator other than the coordination operator in the data lake is the first flag, by using the operator other than the coordination operator; the second flag bit is the flag bit of the operator content in the data lake, used to indicate whether the coordination operator has completed the update.

8. The apparatus according to claim 7, characterized in that, The data lake includes operators for parallel processing, and the apparatus further includes: A determining unit is used to determine the key value in the data structure change event based on the target parallelism of the parallel processing operators in the data lake; the target parallelism is used to indicate the number of parallel processing units in the parallel processing operators. Correspondingly, the construction module includes: The first construction unit is used to construct a coordination operator event based on the key value and the data change information, and to send the coordination operator event to the operators in the data lake.

9. A data synchronization device, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing the computer program to cause the device to perform the data synchronization method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the data synchronization method as described in any one of claims 1 to 6.

Images