A distributed incremental data synchronization method and device based on Flink

Abstract

The present invention discloses a distributed incremental data synchronization method and device based on Flink, which belongs to the field of network application technology, including: determining the block boundary according to the preset block key and block size in the job manager, and constructing a sharded database query statement; determining the checksum calculation rule of the block, and generating a parameterized database query statement; storing database information in the table manager, and obtaining the checksum data storage table; calculating the initial checksum of the block, and writing all of them into the checksum data storage table; receiving the block information issued by the job manager in the task manager; initializing and executing the incremental data synchronization task, calculating the latest checksum of the target block to compare with its initial checksum, and performing the incremental data synchronization operation according to the comparison result. The present application can obtain the data changes in the table based on the specified column without the need for the database to open the incremental field, which greatly improves the timeliness of data changes in the data integration scenario.

Description

Technical Field

[0001] The present invention relates to the field of network application technologies, and in particular to a distributed incremental data synchronization method and device based on Flink. Background Art

[0002] In the field of big data integration and processing, data synchronization is a key link to ensure data consistency and timeliness. Traditional data integration frameworks usually adopt an incremental synchronization mechanism based on timestamps or unique identifiers (IDs). The basis of this mechanism is the existence of clear incremental fields in the data source. New or changed data records are identified and synchronized by comparing the values ​​of these fields.

[0003] However, in actual applications, not all data sources have such incremental fields. When the data source lacks a clear timestamp or unique identifier, the traditional incremental synchronization method cannot work effectively. In this case, in order to maintain data consistency, the data integration framework often has to adopt a full synchronization method, that is, retransmitting the entire data set each time synchronization occurs. This approach is not only inefficient, but also wastes a lot of network resources and computing resources.

[0004] In addition, even if the data source has incremental fields, traditional synchronization methods face another challenge: they are unable to accurately identify the type of data changes, especially when data is updated or deleted. Traditional synchronization methods often cannot distinguish between these two types of changes, resulting in data inconsistencies during the synchronization process. Traditional synchronization methods are even more powerless for deletion operations because they lack an effective mechanism to identify and synchronize deleted data records.

[0005] To solve these problems, the industry has been exploring more efficient and accurate data synchronization methods. Among them, a data synchronization method based on hash code and cyclic redundancy check (CRC) has gradually emerged. This method generates a unique identifier for the data by calculating the hash code or CRC value of the data record, and then quickly identifies and synchronizes the changed data records by comparing these unique identifiers.

[0006] However, although this method is feasible in theory, it still faces many challenges in practical applications, such as how to ensure the accuracy and stability of hash codes or CRC values, how to efficiently calculate and store these unique identifiers, and how to implement this data synchronization method based on hash codes and CRC values ​​in actual data integration scenarios. These are all issues that need to be solved urgently. Summary of the invention

[0007] The purpose of the present invention is to provide a distributed incremental data synchronization method and device based on Flink, so as to solve the problem that the prior art data integration framework cannot identify the change records of rows when realizing incremental data synchronization, and can only perform full data synchronization when data is deleted.

[0008] To achieve the above objectives, this application adopts the following technical solutions:

[0009] The present invention provides a distributed incremental data synchronization method based on Flink, comprising the following steps:

[0010] Determine the boundary of each block in the job manager according to the preset block key and block size, and construct a sharded database query statement according to the boundary;

[0011] Determine the block verification and calculation rules according to the target verification algorithm, and generate parameterized database query statements corresponding to the verification and calculation rules;

[0012] Initialize a table manager to store database information related to each incremental data synchronization in the table manager, and obtain a checksum data storage table in its metadata database for storing a block index and its corresponding checksum;

[0013] Calculate the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and write all the obtained initial checksum and its block index into the check data storage table;

[0014] Receiving, in the task manager, the block information including the target block index issued by the job manager, and extracting the initial checksum corresponding to the target block from the check data storage table according to the target block index;

[0015] Initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare with its initial checksum, and perform the incremental data synchronization operation according to the comparison result.

[0016] Preferably, the step of determining the boundary of each block according to a preset block key and block size comprises:

[0017] Constructing a first database query statement for querying the minimum value and the maximum value of the partition key according to a preset partition key, and executing the first database query statement to obtain the minimum value and the maximum value of the partition key;

[0018] The estimated row value of the source table in the source database for incremental data synchronization is obtained, and the boundary of each block is determined according to a preset block size, a minimum value and a maximum value of the block key, and the estimated row value.

[0019] Preferably, the target verification algorithm includes a target first verification algorithm and a target second verification algorithm;

[0020] Before determining the block verification and calculation rules according to the target verification algorithm, it also includes:

[0021] Determine whether there is a first algorithm parameter in the pre-configured task parameters; if so, use the value of the first algorithm parameter as the target first verification algorithm; if not, query the algorithms supported by the source database among the built-in multiple first verification algorithms, and use the first algorithm supported by the queried source database as the target first verification algorithm;

[0022] Determine whether there is a second algorithm parameter in the task parameters. If so, use the value of the second algorithm parameter as the target second verification algorithm. If not, obtain multiple built-in second verification algorithms and filter out the target second verification algorithm from the multiple built-in second verification algorithms according to the data synchronization filtering conditions.

[0023] Preferably, the target second verification algorithm includes a bitwise XOR algorithm and a user-defined second verification algorithm;

[0024] Determining the block verification and calculation rules according to the target verification algorithm includes:

[0025] Determining whether a check code generated according to a target first check algorithm is an integer;

[0026] When the check code is an integer and the target second check algorithm is a bitwise XOR algorithm, performing an XOR operation on the check code and converting the operation result into a lowercase hexadecimal string;

[0027] When the check code is not an integer and the target second check algorithm is a bitwise XOR algorithm, segment the expression of the check code according to its width and concatenate the segmentation results and convert them into a lowercase hexadecimal string;

[0028] When the check code is an integer and the target second check algorithm is the custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the output is converted into a hexadecimal string, and the target second check algorithm is called to calculate the converted hexadecimal string;

[0029] When the check code is not an integer and the target second check algorithm is a custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the target second check algorithm is called to calculate the output.

[0030] Preferably, the obtaining of a checksum data storage table in the metadata database for storing the block index and its corresponding checksum includes:

[0031] Determine whether there is a check data storage table for storing block indexes and their corresponding checksums in the metadata database of the table manager. If not, construct a check data storage table based on the block information, wherein the check data storage table is used to store the block indexes and all their corresponding checksums.

[0032] Preferably, the calculating the initial checksum of each block based on the sharded database query statement and the parameterized database query statement comprises:

[0033] The parameterized database query statement is instantiated according to the target check column and the sharded database query statement to obtain multiple second database query statements for calculating the block checksum, and all the second database query statements are executed to obtain the initial checksum of each block.

[0034] Preferably, the step of performing the incremental data synchronization operation according to the comparison result includes:

[0035] When the latest checksum is equal to the initial checksum, there is no need to perform an incremental data synchronization operation;

[0036] When the latest checksum is not equal to the initial checksum, the data sets corresponding to the target block in the source table and the target table are obtained, and the intersection and difference of the two data sets are calculated, and data update, insertion or deletion operations are performed according to the calculation results.

[0037] A distributed incremental data synchronization device based on Flink, comprising:

[0038] A construction module, used to determine the boundary of each block according to a preset block key and block size in the job manager, and to construct a sharded database query statement according to the boundary;

[0039] A generation module, used to determine the verification and calculation rules of the blocks according to the target verification algorithm, and generate parameterized database query statements corresponding to the verification and calculation rules;

[0040] A storage module, used for initializing a table manager to store database information related to each incremental data synchronization in the table manager, and obtaining a checksum data storage table in its metadata database for storing a block index and its corresponding checksum;

[0041] A calculation module, used for calculating the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and writing all the obtained initial checksum and its block index into the check data storage table;

[0042] An extraction module, configured to receive, in a task manager, the block information including the target block index issued by the job manager, and extract, according to the target block index, an initial checksum corresponding to the target block from the check data storage table;

[0043] The synchronization module is used to initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare with its initial checksum, and perform the incremental data synchronization operation according to the comparison result.

[0044] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a distributed incremental data synchronization method based on Flink as described in any one of the above.

[0045] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a distributed incremental data synchronization method based on Flink as described in any one of the above when executed.

[0046] This application has the following beneficial effects:

[0047] This application can obtain data changes in the table based on specified columns without enabling incremental fields in the database, including identifying deleted rows, without synchronizing the entire table. This greatly improves the timeliness of data changes in data integration scenarios, especially for storage tables with large data volumes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of a distributed incremental data synchronization method based on Flink provided in an embodiment of the present application;

[0049] Figure 2 It is a structural diagram of a distributed incremental data synchronization device based on Flink provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of an electronic device for implementing a distributed incremental data synchronization method based on Flink provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the technical solution of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] like Figure 1 As shown, the present application provides a distributed incremental data synchronization method based on Flink, comprising the following steps:

[0053] S110, determining the boundary of each block according to a preset block key and block size in the job manager, and constructing a sharded database query statement according to the boundary;

[0054] S120, determining the block verification and calculation rules according to the target verification algorithm, and generating parameterized database query statements corresponding to the verification and calculation rules;

[0055] S130, initializing a table manager to store database information related to each incremental data synchronization in the table manager, and obtaining a checksum data storage table in its metabase for storing a block index and its corresponding checksum;

[0056] S140, calculating the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and writing all the obtained initial checksum and its block index into the check data storage table;

[0057] S150, receiving the block information including the target block index sent by the job manager in the task manager, and extracting the initial checksum corresponding to the target block from the check data storage table according to the target block index;

[0058] S160, initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare with its initial checksum, and perform the incremental data synchronization operation according to the comparison result.

[0059] This application is implemented based on Flink, and mainly applies three components in Flink, namely the job manager JobManager, the table manager TableManager and the task manager TaskManager. Among them, Flink is an open source, distributed, high-performance, high-availability, high-throughput and low-latency big data processing engine, which adopts a master-slave architecture, including a master node JobManager and multiple slave nodes TaskManager. JobManager is responsible for coordinating and managing the execution of jobs, including task scheduling, resource allocation and status management, etc. TaskManager is responsible for specific task execution, including data reception, processing and output, etc. TableManager is a component in the Flink Table & SQL module, responsible for managing the metadata of the table.

[0060] Before executing the incremental data synchronization task, you need to create the task on the client and submit it. Specifically, configure the task parameters, which include the source database name db and the target database name, the source table name tb1 and the target table name, the chunk size chunk_size, the chunk key chunk_col, the target check column crc_col, the first algorithm parameter function, the second algorithm parameter algorithm, and the data synchronization filter condition where. The chunk key is the column name used for chunking, the target check column is the column in the chunk used to calculate the cyclic redundancy checksum, the first algorithm parameter and the second algorithm parameter are used to specify the target first check algorithm and the target second check algorithm, respectively. They are not required items like where. Then add the parameters for starting the Flink task, mainly specifying the environment required for the task to run in the Flink task configuration file flink.yaml, such as yarn or standalone. After all configurations are completed, submit the incremental data synchronization task.

[0061] Furthermore, the boundary of each block is determined according to the preset block key and block size, including:

[0062] Constructing a first database query statement for querying the minimum value and the maximum value of the block key according to the preset block key, and executing the first database query statement to obtain the minimum value and the maximum value of the block key;

[0063] Get the estimated row value of the source table in the source database for incremental data synchronization, and determine the boundary of each block based on the preset block size, the minimum and maximum values ​​of the block key, and the estimated row value.

[0064] After the client submits the task, the JobManager performs a series of operations to manage and coordinate the execution of the task. Specifically, it first constructs the first database query statement based on the chunk_col in the task parameter transmitted by the client, which is the pre-configured chunk_col. Taking the block key as column col1 as an example, the first database query statement is (select min(col1), max(col1) from tb1), which is used to query the maximum and minimum values ​​of column col1. At the same time, the estimated number of source table rows is obtained by executing the EXPLAIN statement or its equivalent form, which depends on the specific database system. For example, when the source database is MySQL, it is "EXPLAINSELECT*FROM table". It should be noted that the "estimated number of rows" returned by the EXPLAIN statement is a predicted value based on statistical information and indexes, which is not always completely accurate. Then, the boundaries of each chunk are calculated based on the pre-configured chunk_size, the minimum and maximum values ​​of the chunk key, and the estimated number of rows row num. For example, the boundary of the first chunk is col1>=1 and col1<1000, which means that the value of col1 in the first chunk must be between 1 (inclusive) and 1000 (exclusive), and the chunk information is stored in a list called chunks. Then, sharded database query statements are constructed based on these boundaries. Sharded database query statements refer to a type of SQL statement that performs query, update, and other operations on tables that have been processed using chunking technology in the database, which can improve the overall performance and resource utilization of the database.

[0065] Furthermore, before determining the block verification and calculation rules according to the target verification algorithm, it also includes:

[0066] Determine whether the first algorithm parameter exists in the pre-configured task parameters. If so, use the value of the first algorithm parameter as the target first verification algorithm. If not, query the algorithms supported by the source database among the built-in multiple first verification algorithms, and use the first algorithm supported by the queried source database as the target first verification algorithm.

[0067] Determine whether there is a second algorithm parameter in the task parameters. If so, use the value of the second algorithm parameter as the target second verification algorithm. If not, obtain multiple built-in second verification algorithms and filter out the target second verification algorithm from the multiple built-in second verification algorithms according to the data synchronization filtering conditions.

[0068] At the same time, JobManager also needs to determine whether the task parameters transmitted by the Client contain the first algorithm parameter [function]. If so, its value is the target first verification algorithm. If not, first obtain the built-in first verification algorithm, including but not limited to CRC32, FNV1A_64, FNV_64, MD5 and SHA1, and then select func('test') finds the algorithms supported by the source database in these functions. The first algorithm supported by the source database is the target first check algorithm. Among them, CRC32 is a cyclic redundancy check algorithm, and its value is called a cyclic redundancy check code. Cyclic redundancy check (CRC) is a channel coding technology used to detect errors that may occur after data transmission or storage. Its principle is to express the information M(X) to be transmitted as a polynomial L, and divide L by a predetermined polynomial G(X). The remainder obtained is the required cyclic redundancy check code. FNV1A_64, FNV_64, MD5 and SHA1 are all hash algorithms. However, in this embodiment, no matter whether the target first check algorithm is a cyclic redundancy check algorithm or a hash algorithm, the value generated by it is uniformly called a check code. It is also necessary to determine whether the task parameter contains a second algorithm parameter [algorithm]. If so, its value is the target second check algorithm. If not, the built-in second check algorithm is also obtained first. The built-in second check algorithm includes a checksum algorithm (CHECKSUM), a bitwise exclusive OR algorithm (BIT_XOR) and a custom second check algorithm (ACCUM).

[0069] Furthermore, the block verification and calculation rules are determined according to the target verification algorithm, including:

[0070] Determining whether a check code generated according to a target first check algorithm is an integer;

[0071] When the check code is an integer and the target second check algorithm is a bitwise XOR algorithm, performing an XOR operation on the check code and converting the operation result into a lowercase hexadecimal string;

[0072] When the check code is not an integer and the target second check algorithm is a bitwise XOR algorithm, the check code expression is split according to its width and the split results are concatenated and converted into a lowercase hexadecimal string;

[0073] When the check code is an integer and the target second check algorithm is not a bitwise XOR algorithm, the check code is concatenated with its accumulated value and then input into the target first check algorithm for calculation to obtain an output, and the output is converted into a hexadecimal string, and the target second check algorithm is called to calculate the converted hexadecimal string;

[0074] When the check code is not an integer and the target second check algorithm is not a bitwise XOR algorithm, the check code is concatenated with its accumulated value and input into the target first check algorithm to calculate the output, and the target second check algorithm is called to calculate the output.

[0075] After the target first verification algorithm and the target second verification algorithm are determined, the verification and calculation rules are determined according to the target first verification code algorithm and the target second verification algorithm.

[0076] Specifically, if the checksum generated according to the target first checksum algorithm is an integer of type "int", and the target second checksum algorithm is BIT_XOR, then the checksum calculation rule is:

[0077] COALESCE(LOWER(CONV(BIT_XOR(CAST("+expr+" AS UNSIGNED)),10,16)),0).

[0078] In this case, the check code expr is first converted into an unsigned integer, and then the BIT_XOR algorithm is called to perform an XOR operation and the XOR result is converted into a hexadecimal string, and then the hexadecimal string is converted to lowercase, and it is ensured that the result is not NULL. Among them, the check code generated according to the target first check algorithm refers to the row checksum calculated by calling the target first check algorithm on the row data in the block.

[0079] If the checksum is not an integer and the target second checksum algorithm is BIT_XOR, the checksum calculation rule is:

[0080] String slices=makeXorSlices(expr,crcWid,optSlice:null);

[0081] result="COALESCE(LOWER(CONV("+slices+")),0)".

[0082] In this case, the expression of the check code is first split according to the CRC width, and then the split results are spliced. Then, the spliced ​​result is converted to a hexadecimal string, and the hexadecimal string is converted to lowercase, while ensuring that the result is not NULL.

[0083] If the target second checksum algorithm is ACCUM and the checksum is an integer of type "int", the checksum calculation rules include:

[0084] String crcFunction=CONV(CAST(func(CONCAT(@crc,expr)) AS UNSIGNED),10, 16).

[0085] In this case, the check code is first concatenated with its accumulated value, and the target first check algorithm is called to calculate the concatenated result to obtain the output result, and then the output result is first converted to an unsigned integer and then converted to a hexadecimal string.

[0086] If the target second checksum algorithm is ACCUM and the checksum is not an integer, the checksum calculation rules include:

[0087] String crcFunction=func(CONCAT(@crc,expr)).

[0088] In this case, the check code is first concatenated with its accumulated value, and then the target first check algorithm is directly called to calculate the concatenated result.

[0089] It should be noted here that in both cases where the target second checksum algorithm is ACCUM, the calculated crcFunction value needs to be embedded in the database expression corresponding to the ACCUM algorithm to generate the checksum. The database expression is COALESCE(RIGHT(MAX("+"@crc:=CONCAT(LPAD(@cnt:=@cnt+1,16,"0"),"+crcFunction+")"+"),"+crcWid+"),0), where @crc represents the cumulative variable of the checksum, which is updated as the row data changes, and @cnt represents the cumulative value of the counter. The product variable is scanned, and the @cnt value is increased by 1 each time a row of data is scanned. The LPAD function is used to convert the @cnt value into a string, and zeros are padded on the left side of the string to make the total length of the string reach 16 characters. Then, the padded string is concatenated with the crcFunction value, and the concatenated result is assigned to the @crc variable. The maximum value function MAX is used to extract the maximum @crc value, and the RIGHT function is called to extract a specified number of characters from the right side of the maximum value string, which is 16 bits here. Finally, the COALESCE function returns the first non-empty value output by the RIGHT function as the checksum of the block.

[0090] At the same time, the table manager TableManager is initialized through the task parameters passed in the JobManager. The TableManager is used to store the database information for each incremental data synchronization operation and check whether there is a checksums data storage table in the TableManager. If not, a checksums table is created. The checksums table is used to record the current synchronized source database name db, table name tb1, chunk number chunk, chunk index chunk_index, chunk lower bound lower_boundary and upper bound upper_boundary, the checksum this_crc of the current chunk, and the number of records in the current chunk this_cnt.

[0091] Furthermore, the initial checksum of each block is calculated based on the sharded database query statement and the parameterized database query statement, including:

[0092] The parameterized database query statement is instantiated according to the target check column and the sharded database query statement to obtain multiple second database query statements for calculating the block checksum, and all the second database query statements are executed to obtain the initial checksum of each block.

[0093] Generate a parameterized database query statement according to the checksum calculation rule. A parameterized database query statement is also called a parameterized query or a parameterized statement. The core is to separate the data input by the user from the rest of the SQL statement, and use parameter placeholders to represent the positions where data needs to be filled. When the query is executed, the database server will first compile and parse the SQL statement, then replace the parameter values provided by the user into the corresponding placeholders, and finally execute the query. For the same query statement, it only needs to be compiled once and can be reused. In this embodiment, after generating the parameterized database query statement, obtain the target check column crc_col in the task parameters, and combine the sharded database query statement to convert the placeholders in the parameterized database query statement into real values, obtaining multiple second database query statements for calculating the block checksum. Execute each second database query statement to obtain the initial checksum of each block. For example, the obtained second database query statement is SELECT / *rows in chunk* / CRC32 (CONCAT_WS('#',`col1`,TRIM(`col2`),`col3`,TRIM(`col4`),`col5`+0)) AS id FROM `db`.`table_1` WHERE `col1`>=split1and `col1`<split2 ORDER BY `col1`. This means querying the table table_1 in the source database db, filtering the values of the boundaries split1 and split2 of the column col1, then concatenating the values of the columns based on the specified columns col1, col2, col3, and col4, and finally calculating the checksum. This checksum is recorded as the initial checksum src_crc. Then, store the initial checksum of each obtained block and its block index in the checksums table.

[0094] The JobManager distributes the block information it generates to each TaskManager. After receiving it, the TaskManager will extract the initial checksum src_crc corresponding to the target block from the checksums table according to the target block index included in the block information transmitted by the JobManager, initialize and execute the incremental data synchronization task, calculate the checksum of the target block after the task is completed, record it as the latest checksum dst_crc, compare it with its src_crc, and perform specific incremental data synchronization operations according to the comparison result. At the same time, store the latest checksum dst_crc in the corresponding position in the checksums table.

[0095] Furthermore, perform incremental data synchronization operations according to the comparison result, including:

[0096] When the latest checksum is equal to the initial checksum, there is no need to perform incremental data synchronization;

[0097] When the latest checksum is not equal to the initial checksum, the data sets corresponding to the target block in the source table and the target table are obtained, and the intersection and difference of the two data sets are calculated, and data update, insertion or deletion operations are performed according to the calculation results.

[0098] If dst_crc is equal to src_crc, it means that the data in the target block has not changed, and the operation is skipped. If dst_crc is not equal to src_crc, the corresponding data sets of the target block in the source table of the source database and the target table of the target database are obtained respectively, and recorded as the original data set and the target data set. Then, the intersection and difference of the original data set and the target data set are calculated, where the intersection is the elements shared by the two data sets, the difference set obtained by subtracting the target data set from the original data set is the data set to be deleted, and the difference set obtained by subtracting the original data set from the target data set is the data set to be inserted. In other words, the data in the original data set but not in the target data set should be deleted. For the elements in the intersection, if their attributes are different in the two sets, an update operation should be performed, and the data in the target data set but not in the original data set should be inserted.

[0099] Compared with the prior art, the present application can obtain data changes in the table based on specified columns without enabling incremental fields in the database, including identifying deleted rows, without the need to synchronize the entire table. This greatly improves the timeliness of data changes in data integration scenarios, especially for storage tables with large amounts of data.

[0100] like Figure 2 As shown, the present application also provides a distributed incremental data synchronization device based on Flink, including:

[0101] A construction module, used to determine the boundary of each block according to a preset block key and block size in the job manager, and to construct a sharded database query statement according to the boundary;

[0102] A generation module is used to determine the verification and calculation rules of the blocks according to the target verification algorithm, and generate parameterized database query statements corresponding to the verification and calculation rules;

[0103] A storage module is used to initialize the table manager to store database information related to each incremental data synchronization in the table manager, and obtain a checksum data storage table in its metadata database for storing block indexes and their corresponding checksums;

[0104] A calculation module is used to calculate the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and write the obtained initial checksum and its block index into the check data storage table;

[0105] An extraction module is used to receive the block information including the target block index sent by the job manager in the task manager, and extract the initial checksum corresponding to the target block from the check data storage table according to the target block index;

[0106] The synchronization module is used to initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare it with its initial checksum, and perform the incremental data synchronization operation based on the comparison result.

[0107] The above-mentioned device can execute the methods provided by all the aforementioned embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above-mentioned methods. For technical details not fully described in this embodiment, please refer to the methods provided by all the aforementioned embodiments of the present invention.

[0108] like Figure 3 As shown, the present application also provides an electronic device, including a memory 301 and a processor 302, wherein the memory 301 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 302 to implement the above-mentioned distributed incremental data synchronization method based on Flink.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0110] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a distributed incremental data synchronization method based on Flink as described above when executed.

[0111] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 301 and executed by the processor 302, and the input interface 305 and the output interface 306 complete the I / O interface transmission of data to complete the present invention. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0112] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a memory 301 and a processor 302. Those skilled in the art may understand that this embodiment is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 307, a network access device, a bus, etc.

[0113] The processor 302 may be a central processing unit (CPU), or other general-purpose processors 302, digital signal processors 302 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 302 may be a microprocessor 302, or the processor 302 may be any conventional processor 302, etc.

[0114] The memory 301 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory 301 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the memory 301 may include both an internal storage unit of the computer device and an external storage device. The memory 301 is used to store computer programs and other programs and data required by the computer device. The memory 301 may also be used to temporarily store in the output device 308. The aforementioned storage media include various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM 303, a random access memory RAM 304, a disk or an optical disk.

[0115] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A distributed incremental data synchronization method based on Flink, characterized in that: The following steps are involved: Determine the boundary of each block in the job manager according to the preset block key and block size, and construct a sharded database query statement according to the boundary; Determine the block checksum calculation rule according to the target checksum algorithm, and generate a parameterized database query statement corresponding to the checksum calculation rule, the target checksum algorithm includes a target first checksum algorithm and a target second checksum algorithm, and the target second checksum algorithm includes a bitwise XOR algorithm and a custom second checksum algorithm; Initialize a table manager to store database information related to each incremental data synchronization in the table manager, and obtain a checksum data storage table in its metadata database for storing block indexes and their corresponding checksums; Calculate the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and write all the obtained initial checksum and its block index into the check data storage table; Receiving, in the task manager, the block information including the target block index issued by the job manager, and extracting, according to the target block index, the initial checksum corresponding to the target block from the check data storage table; Initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare with its initial checksum, and perform the incremental data synchronization operation according to the comparison result; The step of determining the block verification and calculation rules according to the target verification algorithm includes: Determining whether a check code generated according to a target first check algorithm is an integer; When the check code is an integer and the target second check algorithm is a bitwise XOR algorithm, performing an XOR operation on the check code and converting the operation result into a lowercase hexadecimal string; When the check code is not an integer and the target second check algorithm is a bitwise XOR algorithm, segment the expression of the check code according to its width and concatenate the segmentation results and convert them into a lowercase hexadecimal string; When the check code is an integer and the target second check algorithm is the custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the output is converted into a hexadecimal string, and the target second check algorithm is called to calculate the converted hexadecimal string; When the check code is not an integer and the target second check algorithm is a custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the target second check algorithm is called to calculate the output.

2. According to a Flink-based distributed incremental data synchronization method according to claim 1, it is characterized in that: Determining the boundary of each block according to the preset block key and block size includes: Constructing a first database query statement for querying the minimum value and the maximum value of the block key according to a preset block key, and executing the first database query statement to obtain the minimum value and the maximum value of the block key; The estimated row value of the source table in the source database for incremental data synchronization is obtained, and the boundary of each block is determined according to a preset block size, a minimum value and a maximum value of the block key, and the estimated row value.

3. According to a Flink-based distributed incremental data synchronization method according to claim 1, it is characterized in that: Before determining the block verification and calculation rules according to the target verification algorithm, it also includes: Determine whether there is a first algorithm parameter in the pre-configured task parameters; if so, use the value of the first algorithm parameter as the target first verification algorithm; if not, query the algorithms supported by the source database among the built-in multiple first verification algorithms, and use the first algorithm supported by the queried source database as the target first verification algorithm; Determine whether there is a second algorithm parameter in the task parameters. If so, use the value of the second algorithm parameter as the target second verification algorithm. If not, obtain multiple built-in second verification algorithms and filter out the target second verification algorithm from the multiple built-in second verification algorithms according to the data synchronization filtering conditions.

4. According to a Flink-based distributed incremental data synchronization method according to claim 1, it is characterized in that: The obtaining of the checksum data storage table for storing the block index and its corresponding checksum in the metadata database includes: Determine whether there is a check data storage table for storing block indexes and their corresponding checksums in the metadata database of the table manager. If not, construct a check data storage table based on the block information, wherein the check data storage table is used to store the block indexes and all their corresponding checksums.

5. According to a Flink-based distributed incremental data synchronization method according to claim 1, it is characterized in that: The calculating the initial checksum of each block based on the sharded database query statement and the parameterized database query statement includes: The parameterized database query statement is instantiated according to the target check column and the sharded database query statement to obtain multiple second database query statements for calculating the block checksum, and all the second database query statements are executed to obtain the initial checksum of each block.

6. According to a Flink-based distributed incremental data synchronization method according to claim 1, it is characterized in that: The performing of the incremental data synchronization operation according to the comparison result includes: When the latest checksum is equal to the initial checksum, there is no need to perform an incremental data synchronization operation; When the latest checksum is not equal to the initial checksum, the data sets corresponding to the target block in the source table and the target table are obtained, and the intersection and difference of the two data sets are calculated, and data update, insertion or deletion operations are performed according to the calculation results.

7. A distributed incremental data synchronization device based on Flink, characterized in that: include: A construction module, used to determine the boundary of each block according to a preset block key and block size in the job manager, and to construct a sharded database query statement according to the boundary; A generation module, used to determine the block verification and calculation rules according to the target verification algorithm, and generate a parameterized database query statement corresponding to the verification and calculation rules, wherein the target verification algorithm includes a target first verification algorithm and a target second verification algorithm, and the target second verification algorithm includes a bitwise XOR algorithm and a custom second verification algorithm; A storage module, used for initializing a table manager to store database information related to each incremental data synchronization in the table manager, and obtaining a checksum data storage table in its metadata database for storing a block index and its corresponding checksum; A calculation module, used for calculating the initial checksum of each block based on the sharded database query statement and the parameterized database query statement, and writing all the obtained initial checksum and its block index into the check data storage table; An extraction module, configured to receive, in a task manager, the block information including the target block index issued by the job manager, and extract, according to the target block index, an initial checksum corresponding to the target block from the check data storage table; A synchronization module is used to initialize and execute the incremental data synchronization task, calculate the latest checksum of the target block to compare it with its initial checksum, and perform the incremental data synchronization operation according to the comparison result; The generation module determines the block verification and calculation rules according to the target verification algorithm, including: Determining whether a check code generated according to a target first check algorithm is an integer; When the check code is an integer and the target second check algorithm is a bitwise XOR algorithm, performing an XOR operation on the check code and converting the operation result into a lowercase hexadecimal string; When the check code is not an integer and the target second check algorithm is a bitwise XOR algorithm, segment the expression of the check code according to its width and concatenate the segmentation results and convert them into a lowercase hexadecimal string; When the check code is an integer and the target second check algorithm is the custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the output is converted into a hexadecimal string, and the target second check algorithm is called to calculate the converted hexadecimal string; When the check code is not an integer and the target second check algorithm is a custom second check algorithm, the check code and its accumulated value are concatenated and input into the target first check algorithm for calculation to obtain an output, and the target second check algorithm is called to calculate the output.

8. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a distributed incremental data synchronization method based on Flink as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to implement a distributed incremental data synchronization method based on Flink as described in any one of claims 1 to 6 when executed.

Images