Data synchronization method, device, computer equipment, storage medium and program product

Abstract

The embodiment of the present application provides a data synchronization method, device, computer equipment, storage medium and program product. The method includes: matching each row of data in the source table with the target table, determining the first target row of data in the source table that successfully matches the target table; judging whether each row of data in the first target row of data meets the preset deletion condition, deleting each row of data in the first target row of data that meets the preset deletion condition, determining the new first target row of data; updating the new first target row of data to the target table. When merging data in a column storage structure, pre-filtering the data that meets the deletion condition after the batch data is successfully matched can effectively avoid the insertion of invalid data and subsequent deletion operations, reduce unnecessary resource waste, and improve data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data synchronization method, device, computer equipment, storage medium and program product. Background Art

[0002] The MERGE statement is an SQL operation used to merge data from a source table into a target table based on specified conditions. It combines the INSERT, UPDATE, and DELETE functions and is often used for data synchronization. The MERGE statement is a row-by-row operation. When processing data, it compares records in the source and target tables row by row, and decides which operation to perform on each row of data based on the matching conditions. Syntactically, MERGE matches tuples in the source and target tables through the ON condition: if the match is successful, UPDATE can be executed to update the data in the target table to make it consistent with the source table; if it does not match, INSERT can be executed to insert new tuples into the target table; if a delete condition is set, the data that meets the delete condition will be deleted after the update is completed.

[0003] In a column storage database, a batch contains multiple rows of data. Inserting all the data in the batch into the table and then selecting the corresponding rows to delete will result in multiple rewriting and compression of the same data block, which in turn causes a large amount of I / O operations and computing overhead. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, apparatus, computer equipment, storage medium and program product to achieve the effect of improving processing efficiency and reducing waste of storage and computing resources.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method, which is applied to a source table and a target table using a column storage structure, and the method includes:

[0006] Match each row of data in the source table with the target table, and determine the first target row of data in the source table that successfully matches the target table;

[0007] Determine whether each row of data in the first target row of data satisfies a preset deletion condition, delete each row of data in the first target row of data that satisfies the preset deletion condition, and determine a new first target row of data;

[0008] Update the new first target row data into the target table.

[0009] In a possible implementation, the method further includes:

[0010] Determine the second target row data in the source table that fails to match the target table;

[0011] Insert the second target row data into the target table.

[0012] In a possible implementation, matching each row of data in the source table with the target table, and determining first target row of data in the source table that successfully matches the target table and second target row of data that fails to match the target table include:

[0013] Compare the first column value of the first preset column in each row of data in the source table with the second column value of the first preset column in each row of data in the target table to obtain a comparison result;

[0014] According to the comparison result, the first target row data and the second target row data are determined.

[0015] In a possible implementation, determining the first target row data and the second target row data according to the comparison result includes:

[0016] When the first column value is equal to the second column value, determining any row of data as the first target row of data that successfully matches the target table;

[0017] In the case that the first column value is not equal to the second column value, any row of data is determined to be second target row data that fails to match the target table.

[0018] In a possible implementation, updating the new first target row data into the target table includes:

[0019] Delete all row data in the target table that successfully matches the first target row data;

[0020] Write the new first target row data into the target table.

[0021] In a possible implementation manner, determining whether each row of data in the first target row of data satisfies a preset deletion condition includes:

[0022] It is determined whether the third column value of the second preset column in each row of data in the first target row of data is equal to the preset value.

[0023] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0024] A matching module is used to match each row of data in the source table with the target table, and determine the first target row of data in the source table that successfully matches the target table;

[0025] A judgment module, used to judge whether each row of data in the first target row of data meets a preset deletion condition, delete each row of data in the first target row of data that meets the preset deletion condition, and determine a new first target row of data;

[0026] The update module is used to update the new first target row data into the target table.

[0027] In a possible implementation, the device further includes an insertion module, which is used to:

[0028] Determine the second target row data in the source table that fails to match the target table;

[0029] Insert the second target row data into the target table.

[0030] In a possible implementation manner, the matching module is further used for:

[0031] Compare the first column value of the first preset column in each row of data in the source table with the second column value of the first preset column in each row of data in the target table to obtain a comparison result;

[0032] According to the comparison result, the first target row data and the second target row data are determined.

[0033] In a possible implementation manner, the matching module is further used for:

[0034] When the first column value is equal to the second column value, determining any row of data as the first target row of data that successfully matches the target table;

[0035] In the case that the first column value is not equal to the second column value, any row of data is determined to be second target row data that fails to match the target table.

[0036] In a possible implementation, the update module is further used for:

[0037] Delete all row data in the target table that successfully matches the first target row data;

[0038] Write the new first target row data into the target table.

[0039] In a possible implementation manner, the determination module is further configured to:

[0040] It is determined whether the third column value of the second preset column in each row of data in the first target row of data is equal to the preset value.

[0041] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0046] The embodiment of the present application provides a data synchronization method, device, computer equipment, storage medium and program product, the method includes matching each row of data in the source table with the target table, determining the first target row of data in the source table that successfully matches the target table; judging whether each row of data in the first target row of data meets the preset deletion condition, deleting each row of data in the first target row of data that meets the preset deletion condition, determining new first target row of data; updating the new first target row of data to the target table. When merging data in a column storage structure, pre-filtering the data that meets the deletion condition after the batch data is successfully matched can effectively avoid the insertion of invalid data and subsequent deletion operations, reduce unnecessary resource waste, and improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A schematic diagram of an application scenario of a data synchronization method provided in one embodiment of the present application;

[0049] Figure 2 A flowchart of a data synchronization method provided in one embodiment of the present application;

[0050] Figure 3 A flowchart of a data synchronization method provided in another embodiment of the present application;

[0051] Figure 4 A flowchart of a data synchronization method provided in another embodiment of the present application;

[0052] Figure 5 A flowchart of a MEGER processing method based on a column storage structure provided in one embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a data synchronization device provided in one embodiment of the present application;

[0054] Figure 7A schematic diagram of the internal structure of a computer device provided in one embodiment of the present application.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0058] The MERGE statement is an SQL statement used in a relational database management system (RDBMS) to merge data from two tables. It is usually used to merge data from one table (source table) into another table (target table) based on matching conditions.

[0059] The MERGE statement is used to combine UPDATE and INSERT statements. Through the MERGE statement, a table (source table) or a subquery is queried based on the join condition of another table (target table). UPDATE is performed on the table that matches the join condition, and INSERT is performed on the table that does not match the join condition. This syntax only requires one full table scan to complete the entire work, and the execution efficiency is higher than INSERT+UPDATE. The MERGE statement can be applied to SQL Server and Oracle databases.

[0060] The MERGE statement is a row-by-row operation. When processing data, it compares the records in the source table and the target table row by row, and decides which operation to perform based on the matching conditions. In terms of syntax, MERGE matches the tuples of the source table and the target table through the ON condition: if the match is successful, you can execute UPDATE to update the data in the target table to make it consistent with the source table; if it does not match, you can execute INSERT to insert new tuples into the target table; sometimes DELETE operations are also performed under matching conditions to remove tuples that are no longer needed.

[0061] In a column store database, data is stored by column, rather than by row in a traditional row store. This storage method has obvious advantages in querying specific columns and data compression, because it can efficiently read and process data sets of specified columns. However, when performing a MERGE DELETE operation, the processing scheme applied by the row store is not applicable to the column store. The operation method of the row store is based on row-by-row processing: after the tuple of the source table and the target table are matched, the tuple is first updated, and then it is determined whether it needs to be deleted.

[0062] In a column-based database, each operation usually processes a batch of data, which contains multiple rows. Since data is stored and compressed by column, random updates and deletions will result in multiple rewrites and compressions of the same data block, which in turn causes a large amount of I / O operations and computing overhead. This row-by-row operation method is not suitable for column-based storage and will result in a waste of resources.

[0063] In addition, if the MERGE DELETE strategy of traditional row storage is used in a column storage environment, the data needs to be scanned again after the update to locate and delete the data that does not need to be inserted. This repeated processing increases the system complexity and performance consumption, because each deletion operation involves rescanning and processing the data block.

[0064] In view of the above technical problems, the embodiment of the present application provides a data synchronization method for solving the problems of resource waste and low efficiency of MERGE statements in column storage structures. Figure 1 In the application environment shown in the figure, the terminal 102 can access the source table and the target table in the server 104 through the network. Both the source table and the target table adopt a column storage structure. The source table and the target table in the server 104 are stored in a data storage system, which can be integrated on the server 104 or placed on the cloud or other network servers.

[0065] The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0066] In one embodiment, Figure 2 As shown in the figure, a data merging method is provided, which is used to synchronize the data in the source table to the target table. Figure 1 Taking the server 104 in the example as an example, the method includes the following steps:

[0067] Step 202, matching each row of data in the source table with the target table, and determining the first target row of data in the source table that successfully matches the target table;

[0068] Step 204, determining whether each row of data in the first target row of data satisfies a preset deletion condition, deleting each row of data in the first target row of data that satisfies the preset deletion condition, and determining a new first target row of data;

[0069] Step 206: Update the new first target row data into the target table.

[0070] Data merging refers to merging data from two or more tables into one table. The target table is the target of the data merge, and the source table includes the data to be merged. In the embodiment of the present application, the source table and the target table use a column storage structure, which vertically divides the relationship table into multiple sub-relationship tables by column. All data in each group of sub-relationship tables after division are stored in the same data block, and each column is stored independently.

[0071] Data merging refers to merging each row of data in the source table into the target table, including two operations: update and insert for each row of data. Update refers to updating the data of the records that already exist in the target table, and insert refers to inserting the data that does not exist in the target table into the target table. Whether a row of data already exists in the target table needs to be determined by matching the source table with the target table through the preset matching conditions. Specifically, the tuples of the source table and the target table can be matched according to the ON condition. If the match is successful, the Update operation is performed, and if the match is unsuccessful, the Insert operation is performed.

[0072] The first target row data refers to each row data in the source table that successfully matches the target table. The first target row data has a unique corresponding row data in the target table. When matching the source table with the target table, the matching is performed through a preset matching condition, wherein the preset matching condition is usually set based on the column information in the relationship table. For example, in one embodiment, the preset matching condition is that the values ​​of the first column are equal. It should be noted that when setting the preset matching condition, it is necessary to ensure that the matching columns between the target table and the source table are unique, that is, a one-to-one match is successful.

[0073] There are usually multiple rows of data in the source table, so there may be multiple first target rows of data in the source table that are successfully matched with the target table. The update operation for each first target row of data is consistent. The embodiment of the present application is only described with one first target row of data. It should be noted that in actual application scenarios, multiple first target rows of data need to be updated.

[0074] The preset deletion condition is used to filter the first target row data and remove the row data that needs to be deleted. By judging whether each first target row data meets the preset deletion condition, if it meets the condition, the first target row data is deleted, that is, it does not need to be updated into the target table. If it does not meet the condition, the first target row data is updated into the target. When the first target row data is batch data, the row data that meets the preset deletion condition is eliminated by the preset deletion condition to obtain new first target row data. Then the update operation is performed to update the new first target row data into the target table.

[0075] It should be noted that deleting row data will not delete the data in the source table, but will not update the data to be deleted into the target table during the update process of the target table.

[0076] In the method provided in the above embodiment, each row of data in the source table is matched with the target table to determine the first target row of data in the source table that successfully matches the target table; it is determined whether each row of data in the first target row of data meets the preset deletion condition, and each row of data in the first target row of data that meets the preset deletion condition is deleted to determine the new first target row of data; and the new first target row of data is updated to the target table. When merging data in a column storage structure, pre-filtering the data that meets the deletion condition after the batch data is successfully matched can effectively avoid the insertion of invalid data and subsequent deletion operations, reduce unnecessary resource waste, and improve data processing efficiency.

[0077] In one embodiment, the method further comprises:

[0078] Determine the second target row data in the source table that fails to match the target table;

[0079] Insert the second target row data into the target table.

[0080] The second target row data refers to each row data in the source table that does not successfully match the target table, and the second target row data is also determined according to the preset matching conditions. Specifically, through the condition matching in the ON clause, the first target row data is successfully matched, and the UPDATE operation is performed to update the column value of the matching row in the target table. If the match is unsuccessful, it is the second target row data, and the INSERT operation is performed to insert the second target row data into the target table.

[0081] In the method provided in the above embodiment, when the match is unsuccessful, the insert operation only requires a full table scan of the source table and the target table to complete all the work, and the execution efficiency is higher than executing INSERT and UPDATE separately.

[0082] In one embodiment, if Figure 3 As shown, each row of data in the source table is matched with the target table, and the first target row of data in the source table that successfully matches the target table and the second target row of data that fails to match the target table are determined, including:

[0083] Step 302, comparing the first column value of the first preset column in each row of data in the source table with the second column value of the first preset column in each row of data in the target table to obtain a comparison result;

[0084] Step 304: Determine the first target row data and the second target row data according to the comparison result.

[0085] The first preset column refers to a specific column in the table for matching. For example, in one embodiment, the first preset column is the first column. When matching is performed according to the first preset column, the value of the first column in each row of data in the source table is compared with the value of the first column in each row of data in the target table. The number of first preset columns set in the preset matching condition can be one or more, and the principle to be followed is that the source table and the target table can be successfully matched one-to-one.

[0086] In addition, in addition to setting the first preset column, the comparison result also needs to be set in the preset matching condition, that is, to clarify the relationship between the first column value and the second column value when the match is successful. Through the comparison result of the first column value and the second column value, and the size condition set in the preset matching condition, it is determined whether each row of data in the source table has a successfully matched row of data in the target table. If so, the row of data in the source table is the first target row of data, otherwise it is the second target row of data.

[0087] In the method provided in the above embodiment, matching is performed by comparing column values, thereby determining the operation type corresponding to each row of data in the source table.

[0088] In one embodiment, if Figure 4 As shown, according to the comparison result, determining the first target row data and the second target row data includes:

[0089] Step 402, when the first column value is equal to the second column value, determining any row of data as the first target row of data that successfully matches the target table;

[0090] Step 404 , when the first column value is not equal to the second column value, determine that any row of data is the second target row of data that fails to match the target table.

[0091] In this embodiment, the first column value and the second column value are equal only when the match is successful, and the corresponding preset matching condition is that the column value of the first preset column is equal. When the first column value and the second column value are equal, the row data is the first target row data, otherwise, it is the second target row data.

[0092] In one embodiment, updating the new first target row data into the target table includes:

[0093] Delete all row data in the target table that successfully matches the first target row data;

[0094] Write the new first target row data into the target table.

[0095] When performing an update operation on the new first target row data, it is necessary to first delete the row data in the target table that successfully matches the first target row data, and then write the new first target row data into the target table. In the above manner, the row data in the target table that successfully matches is deleted, and then the new first target row data that has been filtered out of the data that meets the preset deletion condition is written into the target table, which can effectively reduce the access to the target table and achieve the same effect as in the prior art.

[0096] In one embodiment, determining whether each row of data in the first target row of data satisfies a preset deletion condition includes:

[0097] It is determined whether the third column value of the second preset column in each row of data in the first target row of data is equal to the preset value.

[0098] The second preset column is used to delete row data, which is specifically judged by whether the third column value of the second preset column is equal to the preset value. In an application scenario, it can be set that when the value of the second preset column is equal to the preset value, the row data needs to be deleted. By setting the second preset column and the preset value in the preset deletion condition, the first target row data is filtered, and the subsequent deletion operation can be canceled.

[0099] In actual application scenarios, data is processed in batches. Therefore, in combination with the ideas in the above embodiments, a MEGER processing method based on a column storage structure is provided, which is applied to synchronize a source table with a target table of a column storage structure. Figure 5 As shown, the method includes:

[0100] S501, obtain the batch to be processed, obtain the batch data to be processed through the sub-plan node, and pass it to the batch merge node for processing;

[0101] S502, classify the operation type, match the batch data with the target table, identify the operation type of each row, determine the first target row data and the second target row data, and process the INSERT and UPDATE operations respectively;

[0102] Specifically, put the first target row data to be updated into update-batch, and put the second target row data to be inserted into insert-batch;

[0103] S503, marking the rows to be deleted. When processing the update-batch, check whether there is a DELETE clause; if so, immediately mark the rows that meet the preset deletion conditions;

[0104] S504, performing a filtering operation to filter out the rows marked as to be deleted, and generating a new update-batch;

[0105] S505, updating the storage unit, and writing the filtered update-batch and insert-batch into the storage unit.

[0106] Next, an example is used to explain in detail the process of the MEGER processing method based on the column storage structure:

[0107] Source Table:

[0108] Row 1: (1,3,5);

[0109] Row 2: (2,4,5);

[0110] Row 3: (3,6,7);

[0111] Row 4: (4,8,3);

[0112] Row 5: (5,9,5);

[0113] Target Table:

[0114] Row 1: (1,2,3);

[0115] Row 2: (2,3,4);

[0116] Row 3: (3,5,6);

[0117] Row 4: (5,7,8);

[0118] Target: Merge the source table data into the target table, update if they match, insert if they do not match, the matching condition is that the first column is equal; among the matching rows, the rows that meet the preset deletion condition (the third column is equal to 5) will be deleted.

[0119] Steps in detail:

[0120] 1. Find the matching rows according to the matching conditions of MERGE:

[0121] The source table (1,3,5) matches the target table (1,2,3);

[0122] The source table (2,4,5) matches the target table (2,3,4);

[0123] The source table (3,6,7) matches the target table (3,5,6);

[0124] The source table (5,9,5) matches the target table (5,7,8);

[0125] 2. Get the values ​​to be updated in the matching rows (load the first target row data into update-batch):

[0126] Update the target table (1,2,3) to (1,3,5);

[0127] Update the target table (2,3,4) to (2,4,5);

[0128] Update the target table (3,5,6) to (3,6,7);

[0129] Update the target table (5,7,8) to (5,9,5);

[0130] Load (1,3,5), (2,4,5), (3,6,7), and (5,9,5) into the update-batch.

[0131] 3. Apply preset deletion conditions to the data to be updated:

[0132] Check update-batch and apply the preset deletion condition (delete the row with number 5 in the third column). Rows (1,3,5), (2,4,5) and (5,9,5) meet the deletion condition and need to be marked for deletion.

[0133] 4. Filter the rows to be deleted:

[0134] Before performing any write operations, the rows that meet the preset deletion conditions are filtered out, and (1,3,5), (2,4,5), and (5,9,5) are removed from the update-batch. At this time, only (3,6,7) remains in the first target row data.

[0135] 5. Perform the merge update operation:

[0136] Delete the values ​​(1,3,5), (2,4,5), (3,6,7) and (5,9,5) that meet the matching conditions from the target table;

[0137] Write the new update-batch to the target table, that is, insert (3,6,7).

[0138] 6. Create an insert batch:

[0139] The second target row data in the unmatched source table is added to the insert-batch. The source table's (4,8,3) has no match and is ready to be inserted.

[0140] 7. Perform the merge insert operation:

[0141] Write the insert-batch to the target table and insert (4,8,3).

[0142] After the above operations are processed, the following is the final result of the target table:

[0143] Row 1: (3,6,7);

[0144] Row 2: (4,8,3);

[0145] By filtering out rows marked for deletion in advance during the MERGE operation, the amount of data that needs to be inserted and subsequently deleted is significantly reduced. This optimization directly improves the data processing speed. Since unnecessary data insertion and deletion operations are reduced, the number of I / O operations in the system is significantly reduced, saving overall storage and computing resources. Through the MERGEDELETE filtering mechanism, this solution simplifies the execution logic of the MERGE operation and eliminates the secondary scanning and deletion steps in the traditional method. This not only reduces the complexity of the operation, but also reduces the cost of development and maintenance, making the system easier to manage and expand.

[0146] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0147] Based on the same inventive concept, an embodiment of the present application also provides a data synchronization device. The implementation solution for solving the problem provided by the device is similar to the implementation solution recorded in the above method. Therefore, the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above and will not be repeated here.

[0148] In one embodiment, Figure 6 As shown, the data synchronization device 600 includes a matching module 601, a judgment module 602 and an update module 603, wherein:

[0149] A matching module 601 is used to match each row of data in the source table with the target table, and determine the first target row of data in the source table that successfully matches the target table;

[0150] A judgment module 602 is used to judge whether each row of data in the first target row of data satisfies a preset deletion condition, delete each row of data in the first target row of data that satisfies the preset deletion condition, and determine a new first target row of data;

[0151] The updating module 603 is used to update the new first target row data into the target table.

[0152] In a possible implementation, the device further includes an insertion module 604, configured to:

[0153] Determine the second target row data in the source table that fails to match the target table;

[0154] Insert the second target row data into the target table.

[0155] In a possible implementation, the matching module 601 is further configured to:

[0156] Compare the first column value of the first preset column in each row of data in the source table with the second column value of the first preset column in each row of data in the target table to obtain a comparison result;

[0157] According to the comparison result, the first target row data and the second target row data are determined.

[0158] In a possible implementation, the matching module 601 is further configured to:

[0159] When the first column value is equal to the second column value, determining any row of data as the first target row of data that successfully matches the target table;

[0160] In the case that the first column value is not equal to the second column value, any row of data is determined to be second target row data that fails to match the target table.

[0161] In a possible implementation, the updating module 603 is further configured to:

[0162] Delete all row data in the target table that successfully matches the first target row data;

[0163] Write the new first target row data into the target table.

[0164] In a possible implementation manner, the determination module 602 is further configured to:

[0165] It is determined whether the third column value of the second preset column in each row of data in the first target row of data is equal to a preset value.

[0166] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0167] Figure 7 This is a schematic diagram of the structure of the computer device provided in this application. Figure 7 As shown, the computer device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0168] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0169] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0170] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0171] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0172] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0173] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0174] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0175] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0176] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0177] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0181] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0182] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method is applied to a source table and a target table using a column storage structure, and the method includes: Matching each row of data in the source table with the target table, and determining a first target row of data in the source table that successfully matches the target table; Determine whether each row of data in the first target row of data satisfies a preset deletion condition, delete each row of data in the first target row of data that satisfies the preset deletion condition, and determine a new first target row of data; The new first target row data is updated into the target table.

2. The method according to claim 1, characterized in that The method further comprises: Determine second target row data in the source table that fails to match the target table; Insert the second target row data into the target table.

3. The method according to claim 2, characterized in that The step of matching each row of data in the source table with the target table to determine first target row of data in the source table that successfully matches the target table and second target row of data that fails to match the target table includes: Compare the first column value of the first preset column in each row of data in the source table with the second column value of the first preset column in each row of data in the target table to obtain a comparison result; The first target row data and the second target row data are determined according to the comparison result.

4. The method according to claim 3, characterized in that The determining the first target row data and the second target row data according to the comparison result includes: In the case where the first column value is equal to the second column value, determining any row of data as first target row of data that successfully matches the target table; In the case that the first column value is not equal to the second column value, the any row of data is determined to be second target row data that fails to match the target table.

5. The method according to claim 3, characterized in that: The updating of the new first target row data into the target table includes: Deleting all row data in the target table that successfully matches the first target row data; The new first target row data is written into the target table.

6. The method according to claim 1, characterized in that The determining whether each row of data in the first target row of data satisfies a preset deletion condition includes: It is determined whether the third column value of the second preset column in each row of data in the first target row of data is equal to a preset value.

7. A data synchronization device, characterized in that: The device comprises: A matching module, used to match each row of data in the source table with the target table, and determine the first target row of data in the source table that successfully matches the target table; a judgment module, configured to judge whether each row of data in the first target row of data satisfies a preset deletion condition, delete each row of data in the first target row of data that satisfies the preset deletion condition, and determine a new first target row of data; An update module is used to update the new first target row data into the target table.

8. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images