Synchronous data table quick repair method and device, computer equipment and storage medium

Abstract

The invention relates to a synchronous data table rapid repairing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a first data table of a reading end, a second data table of a writing end and a data table comparison strategy; based on the data table comparison strategy, respectively performing data fragmentation and fragmentation comparison on the first data table and the second data table to obtain a fragmentation comparison result of the first data table and the second data table; if inconsistent data fragments exist in the fragment comparison result, whether incremental synchronization is carried out on the first data table and the second data table or not is judged, and if not, data restoration is carried out on the data fragments of the writing end based on the data fragments of the reading end till the data of the inconsistent data fragments are consistent. By adopting the method, the problems of long time consumption of data comparison and low data recovery efficiency in the prior art can be solved, and automatic recovery of the synchronous data table is realized.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, device, computer equipment, and storage medium for quickly repairing a synchronized data table. Background Art

[0002] With the rapid development of information technology, big data, and domestic database technologies, more and more scenarios require synchronizing data from one database (read-side) to another (write-side). The read and write ends may use different database software and may be deployed on different servers or even in different computer rooms.

[0003] Existing log-based real-time synchronization solutions are complex in their technical principles and rely heavily on external dependencies, such as the log retention period on the source side. This makes it easy for data inconsistencies to occur between the reader and the writer. In such cases, timely detection and automatic remediation are crucial.

[0004] Traditional technologies typically use statistical or full-text comparison techniques to check data consistency between the reader and the writer. While statistical comparison techniques are simple in principle, they suffer from low accuracy. While full-text comparison techniques can improve accuracy, they are inefficient and costly. Furthermore, in the event of data inconsistencies, further notifications to maintenance personnel via SMS or email are required, potentially leading to delayed data repairs and impacting business continuity. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for quickly repairing synchronized data tables that can improve the efficiency of synchronized data table comparison and data repair in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for quickly repairing a synchronized data table, comprising: Obtaining a first data table on the read side, a second data table on the write side, and a data table comparison strategy; Based on the data table comparison strategy, data sharding and shard comparison are performed on the first data table and the second data table respectively, to obtain shard comparison results of the first data table and the second data table; If there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization. If not, repair the data on the write-end data shard based on the read-end data shard until the inconsistent data shards are consistent.

[0007] In one embodiment, the data table comparison strategy includes: First table structure metadata of the first data table and second table structure metadata of the second data table are obtained, and field mapping is performed based on the first table structure metadata and the second table structure metadata to obtain first fields involved in comparison in the first data table and the second data table.

[0008] In one embodiment, the data table comparison strategy further includes: Obtain all time type fields in the first data table and / or the second data table; Filtering all the time type fields to obtain a second field containing index information, where the second field is a segmentation column for data sharding; Based on the first field and the second field, respectively, data sharding is performed on the first data table and the second data table, and a first checksum value of each data shard in the first data table and a second checksum value of each data shard in the second data table are calculated; The first check value and the second check value of each corresponding data shard are compared to obtain a shard comparison result of the first data table and the second data table.

[0009] In one embodiment, sharding the first data table and the second data table based on the first field and the second field, and calculating a first checksum for each data shard in the first data table and a second checksum for each data shard in the second data table includes: Based on the second field, data sharding is performed on the first data table and the second data table respectively to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table; Obtaining a time point included in each first data shard in the first data shard set, concatenating the first fields corresponding to each time point according to the metadata order of the first table structure to obtain a first long string for each time point; performing a hash calculation based on the first long string for each time point and summing the results to obtain the first check value; Obtain the time points contained in each second data shard in the second data shard set, splice the first fields corresponding to each time point in the order of the second table structure metadata to obtain the second long string for each time point; perform hash calculation and sum based on the second long string of each time point to obtain the second check value.

[0010] In one embodiment, before obtaining all time type fields in the first data table and / or the second data table, the method further includes: Determine whether the user terminal specifies a third field, where the third field is a partitioning column for data partitioning; if so, perform data partitioning on the first data table and the second data table based on the third field and the first field, respectively.

[0011] In one embodiment, the data table comparison strategy further includes: Obtain all numeric fields in the first data table and / or the second data table; A fourth field containing index information is obtained by screening all the numeric type fields, where the fourth field is a segmentation column used for data sharding.

[0012] In one embodiment, the method further comprises: If the shard comparison result shows inconsistent data shards, determining whether the first data table and the second data table are undergoing incremental synchronization; If so, obtain the number of comparisons of the shard comparison result in real time, and determine whether the number of comparisons is greater than the set retry comparison threshold; If it is greater, data repair is performed on the data shards on the write side based on the data shards on the read side until the inconsistent data shards are consistent; otherwise, data sharding and shard comparison are performed on the first data table and the second data table respectively based on the data table comparison strategy until the number of comparisons is greater than the retry comparison threshold.

[0013] In a second aspect, the present application further provides a device for quickly repairing a synchronized data table, comprising: A data reading module, configured to obtain a first data table on a reading end, a second data table on a writing end, and a data table comparison strategy; The comparison and repair module is used to perform data sharding and shard comparison on the first data table and the second data table respectively based on the data table comparison strategy to obtain the shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, data repair is performed on the data shards on the write end based on the data shards on the read end until the data of the inconsistent data shards are consistent.

[0014] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the content of the above-mentioned first aspect when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the content of the above-mentioned first aspect when executed by a processor.

[0016] The above-mentioned method, device, computer equipment and storage medium for rapid repair of synchronized data tables obtain the first data table on the read side, the second data table on the write side and the data table comparison strategy; based on the data table comparison strategy, the first data table and the second data table are respectively subjected to data sharding and shard comparison to obtain the shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, data repair is performed on the data shards on the write side based on the data shards on the read side until the data of the inconsistent data shards are consistent, thereby solving the problems of long data comparison time and low data repair efficiency in the prior art and realizing automatic repair of synchronized data tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A diagram illustrating an application environment of a method for quickly repairing a synchronized data table according to an embodiment; Figure 2 Schematic diagram of a process for quickly repairing a synchronized data table according to an embodiment; Figure 3 1 is a flow chart of a data table comparison strategy in one embodiment; Figure 4 Schematic diagram of the process of data sharding and check value calculation in one embodiment; Figure 5 A structural block diagram of a device for quickly repairing a synchronized data table according to an embodiment; Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0020] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote quantitative limitations and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0021] The method for quickly repairing a synchronization data table provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers, including various database service resources and data warehouse systems.

[0022] When the read and write ends are different database software called on the same terminal, the server obtains the first data table of the read end, the second data table of the write end, and the data table comparison strategy when executing the data table synchronization task of the read and write ends; based on the data table comparison strategy, the first data table and the second data table are respectively subjected to data sharding and shard comparison to obtain the shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, the data shards of the write end are repaired based on the data shards of the read end until the inconsistent data shards are consistent.

[0023] Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0024] In an exemplary embodiment, Figure 2 As shown, a method for quickly repairing a synchronized data table is provided, which is applied to Figure 1 The server side in the example is used to illustrate, including the following steps 201 to 203. Among them: Step 201: Obtain a first data table on the reading end, a second data table on the writing end, and a data table comparison strategy.

[0025] Specifically, the first data table on the reading end and the second data table on the writing end are synchronized data tables. The first data table and the second data table on both ends are obtained in real time to determine whether there is a cached data table comparison strategy on the server. If so, the cached data table comparison strategy is loaded; otherwise, a new data table comparison strategy is adaptively generated.

[0026] Preferably, the adaptively generated new data table comparison strategy is automatically cached to the server.

[0027] Step 202: Based on the data table comparison strategy, data sharding and shard comparison are performed on the first data table and the second data table respectively to obtain shard comparison results of the first data table and the second data table.

[0028] Specifically, the data table comparison strategy includes a data sharding method and a sharding comparison method. The first data table and the second data table are sharded using the same data sharding method, and then the sharding results of the two data tables are compared according to the sharding comparison method to obtain the sharding comparison results of the first data table and the second data table.

[0029] Step 203: If there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization; if not, repair the data on the write-end data shard based on the read-end data shard until the inconsistent data shards are consistent.

[0030] Specifically, based on the shard comparison results, determine whether there are inconsistent data shards in the two data tables. If so, further determine whether the two data tables are undergoing CDC incremental synchronization. If the two data tables are undergoing CDC incremental synchronization, it indicates that there may be replication delay between the two data tables. Only after the CDC incremental synchronization is completed can the data shards on the write side be repaired based on the data shards on the read side of the inconsistent data shards until the data in the inconsistent data shards are consistent.

[0031] In the above-mentioned method for quickly repairing synchronized data tables, a first data table on the read side, a second data table on the write side, and a data table comparison strategy are obtained; based on the data table comparison strategy, data sharding and shard comparison are performed on the first data table and the second data table respectively to obtain shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, data repair is performed on the data shards on the write side based on the data shards on the read side until the data of the inconsistent data shards are consistent, thereby solving the problems of long data comparison time and low data repair efficiency in the prior art and realizing automatic repair of synchronized data tables.

[0032] In an exemplary embodiment, the data table comparison strategy includes: First table structure metadata of the first data table and second table structure metadata of the second data table are obtained, and field mapping is performed based on the first table structure metadata and the second table structure metadata to obtain first fields involved in comparison in the first data table and the second data table.

[0033] In detail, since the read side does not necessarily synchronize all the data in a data table to the write side, before repairing the data of two data tables that have a data synchronization relationship, it is necessary to map the fields of the two data tables according to the table structure metadata of the data tables, obtain the first field involved in the comparison in the two data tables, and complete the alignment of the data table fields involved in the comparison on both the read and write sides.

[0034] In this embodiment, through field mapping of table structure metadata, the fields that need to be compared in the two data tables are accurately obtained, thereby improving the data comparison accuracy.

[0035] In an exemplary embodiment, Figure 3 As shown, the data table comparison strategy further includes the following steps 301 to 304. Among them: Step 301: Acquire all time type fields in the first data table and / or the second data table.

[0036] Time fields include create, update, and modify. Most database data transitions from "hot data" to "cold data" over time. Once data becomes "cold," it becomes less likely to be modified, reducing the likelihood of data inconsistencies. Therefore, sharding data based on time fields in a table can improve the efficiency of data comparison and repair.

[0037] Step 302: Filter all the time type fields to obtain a second field containing index information, where the second field is a partition column used for data sharding.

[0038] Step 303: Based on the first field and the second field, the first data table and the second data table are respectively partitioned, and a first check value for each data partition in the first data table and a second check value for each data partition in the second data table are calculated.

[0039] For example, if the second field of the index information is an update field, the data sharding column, i.e., the sharding basis, is the latest update time. Data sharding is performed based on the latest update time of all first fields involved in the comparison on both the read and write sides. If the second field of the index information is a create field, the data sharding column is the creation time. Data sharding is performed based on the creation time of all first fields involved in the comparison on both the read and write sides. Furthermore, a first checksum value is calculated for each data shard in the first data table and a second checksum value is calculated for each data shard in the second data table.

[0040] Step 304: Compare the first check value and the second check value of each corresponding data slice to obtain a slice comparison result of the first data table and the second data table.

[0041] Specifically, if the first checksum and the second checksum of the data shards at both ends of the read and write operations are exactly the same, it indicates that the data shards at both ends of the read and write operations are consistent; otherwise, the data shards are inconsistent.

[0042] In this embodiment, data sharding is performed on the two data tables by utilizing the second field of the time type and the second fields involved in the comparison in the two tables, and the check value of each data shard in the two tables is calculated. The check values ​​of the two corresponding data shards are compared to obtain the shard comparison results of the first data table and the second data table. When large-scale data inconsistency triggers data repair, the inconsistent data can be concentrated in a small number of shards, which greatly reduces the range of data affected during data repair, thereby improving data repair efficiency.

[0043] In one embodiment, Figure 4As shown, step 303 performs data sharding on the first data table and the second data table based on the first field and the second field, and calculates the first check value of each data shard in the first data table and the second check value of each data shard in the second data table, specifically including the following steps 401 to 403.

[0044] Step 401: Based on the second field, data sharding is performed on the first data table and the second data table respectively to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table.

[0045] Specifically, based on the second field of the time type, the first data table and the second data table are sharded to obtain corresponding first and second data shard sets. Taking the first data shard set as an example, the size of each first data shard contained therein can be set as needed. For example, based on the latest update time of the first field involved in the comparison, a shard is generated every 10 minutes to obtain the first data shard set. The shards of the second data shard set are the same size as those of the first data shard set.

[0046] Step 402: Obtain the time points contained in each first data shard in the first data shard set, concatenate the first fields corresponding to each time point in the order of the first table structure metadata to obtain the first long string for each time point; perform hash calculation and sum based on the first long string for each time point to obtain the first check value.

[0047] Step 403: Obtain the time points contained in each second data shard in the second data shard set, concatenate the first fields corresponding to each time point in the order of the metadata of the second table structure to obtain the second long character string for each time point; perform hash calculation and sum based on the second long character string of each time point to obtain the second check value.

[0048] For steps 402 and 403, the processing process for the first data shard set and the second data shard set is the same. Taking the first data shard set as an example, specifically, for one of the 10-minute first data shards, the first field contained in the field corresponding to the latest update time of each item in the shard, i.e., the data involved in the comparison, is concatenated according to the metadata order of the first table structure to obtain the first long string corresponding to each latest update time field. An MD5 hash calculation is performed on each first long string to obtain a check value for each first long string. The check values ​​of each long string in the first data shard are summed to obtain the first check value of the data shard. Similarly, the second check value of each second data shard in the second data shard set is obtained.

[0049] In this embodiment, by splicing the first field at each time point in the data shard, calculating the hash value of the spliced ​​long string, and summing the hash values ​​of all long strings in the data shard, the check value of the data shard is obtained, thereby improving the accuracy of the data shard comparison results and thus improving the sharding efficiency and repair efficiency.

[0050] In an exemplary embodiment, before step 301 of obtaining all time type fields in the first data table and / or the second data table, the following contents are also included: Determine whether the user terminal specifies a third field, where the third field is a partitioning column for data partitioning; if so, perform data partitioning on the first data table and the second data table based on the third field and the first field, respectively.

[0051] The third field may be a time type field specified by the user terminal, or may be another type of field.

[0052] In this embodiment, before adaptively obtaining the time type field to determine the splitting column for data sharding, it is determined whether the user specifies the splitting column, thereby improving the matching degree between the comparison and repair process and the user's needs and improving the comparison and repair efficiency.

[0053] In an exemplary embodiment, the data table comparison strategy also includes: obtaining all numeric type fields in the first data table and / or the second data table; filtering out a fourth field containing index information from all numeric type fields, wherein the fourth field is a split column used for data sharding.

[0054] Specifically, in the data table comparison strategy, the time type field is the preferred field for the split column. If the time type field does not exist in the first data table or the second data table, or if the time type field does not have index information, all numeric type fields in the data table are obtained and filtered to obtain a fourth field with index information. This field is used as the split column for the data sharding.

[0055] Furthermore, if a fourth field of a numeric type is selected as a split column for data sharding, the data table comparison strategy further includes the following: Based on the first field and the fourth field, the first data table and the second data table are respectively partitioned, and a third check value of each data partition in the first data table and a fourth check value of each data partition in the second data table are calculated.

[0056] In detail, taking the first data table as an example, the size of the data shard is set according to actual needs. Assuming that the numeric type field is the id field, a shard is made for every 10,000 ids according to the id field to obtain the third data shard set of the first data table, and the third data shard set includes at least one third data shard. In each third data shard, the first field corresponding to each id is spliced ​​in the order of the metadata of the first table structure to obtain the third long string of each id, and a hash calculation and summation are performed based on the third long string of each id to obtain the third check value of the third data shard. Similarly, the fourth check value of each fourth data shard in the fourth data shard set corresponding to the second data table is calculated. The third check value and the fourth check value of the corresponding data shards are compared to obtain the shard comparison result of the first data table and the second data table.

[0057] In this embodiment, when the time type field is missing, data segmentation and segmentation comparison are performed using the numeric type field to ensure that the data table comparison strategy of this application is compatible with and supports all types of data tables, thereby realizing the comparison and repair of synchronized data tables.

[0058] In an exemplary embodiment, in step 203, it is determined whether the first data table and the second data table are in incremental synchronization, and the method further includes the following: If there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization; if so, obtain the comparison times of the shard comparison result in real time, and determine whether the comparison times are greater than the set retry threshold; if so, perform data repair on the data shards on the write end based on the data shards on the read end until the data of the inconsistent data shards are consistent; otherwise, re-shard and compare the data on the first data table and the second data table based on the data table comparison strategy until the comparison times are greater than the retry comparison threshold.

[0059] In this embodiment, in the process of determining whether the first data table and the second data table are undergoing incremental synchronization, the synchronization replication delay of the first data table and the second data table is limited to an estimated time by setting a retry count threshold, thereby reducing unnecessary shard comparison time. When the number of comparisons is greater than the retry count threshold, data repair is performed on the data shard on the write side based on the data shard on the read side, thereby improving the efficiency of the comparison and repair process.

[0060] In a preferred embodiment, a method for quickly repairing a synchronized data table is provided, taking a time type field as a segmentation column as an example, and specifically including the following contents: Step 1: Obtain the first data table of the read end and the second data table of the write end.

[0061] Step 2: Try to load the cached data table comparison strategy. If it exists, obtain the valid data table comparison strategy. Otherwise, execute steps 3 to 10 to complete the adaptive generation of the data table comparison strategy and the comparison and repair of the data table.

[0062] Step 3: Obtain the first table structure metadata of the first data table and the second table structure metadata of the second data table, and perform field mapping based on the first table structure metadata and the second table structure metadata to obtain the first field involved in the comparison in the first data table and the second data table.

[0063] Step 4: Determine whether the user terminal specifies the third field for data sharding. If so, use the third field as the sharding column. Otherwise, obtain all time type fields in the first data table and / or the second data table, and filter out the second field with index information from all time type fields as the sharding column.

[0064] Step 5: Based on the second field, perform data sharding on the first data table and the second data table respectively to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table.

[0065] Step 6: Obtain the time points contained in each first data shard in the first data shard set, and concatenate the first fields corresponding to each time point in the order of the first table structure metadata to obtain the first long string of each time point; perform hash calculation and sum based on the first long string of each time point to obtain the first check value of the first data shard.

[0066] Step 7: Obtain the time points contained in each second data shard in the second data shard set, and concatenate the first fields corresponding to each time point in the order of the metadata of the second table structure to obtain the second long string for each time point; perform hash calculation and sum based on the second long string of each time point to obtain the second check value of the second data shard.

[0067] Step 8: Compare the first checksum and second checksum of each corresponding data shard to obtain the shard comparison results of the first data table and the second data table. If the first checksum and the second checksum are exactly the same, the data shards are consistent and the process ends. Otherwise, the data shards are inconsistent and step 9 is executed.

[0068] Step 9: Determine whether the first and second data tables are undergoing incremental synchronization. If not, repair the data on the write-side data shards based on the read-side data shards until the inconsistent data shards are consistent, and the process ends. If so, obtain the comparison count of the shard comparison results in real time and execute step 10.

[0069] Step 10: Determine whether the number of comparisons exceeds the set retry threshold. If so, repair the data on the write-end data shards based on the read-end data shards until the inconsistent data shards are consistent, and the process ends. Otherwise, re-shard and compare the data on the first and second data tables based on steps 3 to 9 until the number of comparisons exceeds the retry threshold.

[0070] Table 1 shows the results of a shard comparison. The results show that the number of records and digest values ​​in data shards 2, 3, and 4 are the same, so the comparison is consistent and no repair is required.

[0071] Table 1

[0072] Data shard 1 may have been deleted on the read side, so it is deleted on the write side according to the shard boundary. Data shard 5 does not exist on the write side. Further determination is made as to whether CDC incremental synchronization is in progress between the read and write sides. If so, the comparison continues. Otherwise, the corresponding data on the read side is written according to the shard boundary.

[0073] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0074] Based on the same inventive concept, embodiments of the present application also provide a synchronized data table rapid repair device for implementing the synchronized data table rapid repair method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more embodiments of the synchronized data table rapid repair device provided below can be found in the limitations of the synchronized data table rapid repair method described above and will not be repeated here.

[0075] In an exemplary embodiment, Figure 5 As shown, a device for quickly repairing a synchronization data table is provided, comprising: a data reading module 501 and a comparison and repair module 502, wherein: The data reading module 501 is used to obtain the first data table of the reading end, the second data table of the writing end, and the data table comparison strategy.

[0076] The comparison and repair module 502 is used to perform data sharding and shard comparison on the first data table and the second data table respectively based on the data table comparison strategy to obtain the shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, data repair is performed on the data shards on the write end based on the data shards on the read end until the data of the inconsistent data shards are consistent.

[0077] In one embodiment, the comparison and repair module 502 is also used to: obtain the first table structure metadata of the first data table and the second table structure metadata of the second data table, and perform field mapping based on the first table structure metadata and the second table structure metadata to obtain the first field participating in the comparison in the first data table and the second data table.

[0078] In one embodiment, the comparison and repair module 502 is also used to: obtain all time type fields in the first data table and / or the second data table; filter out the second field with index information from all time type fields, and the second field is a partitioning column used for data sharding; based on the first field and the second field, perform data sharding on the first data table and the second data table respectively, and calculate the first check value of each data shard in the first data table and the second check value of each data shard in the second data table; compare the first check value and the second check value of each corresponding data shard to obtain the sharding comparison result of the first data table and the second data table.

[0079] In one embodiment, the comparison and repair module 502 is also used to: based on the second field, respectively perform data sharding on the first data table and the second data table to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table; obtain the time points contained in each first data shard in the first data shard set, and splice the first fields corresponding to each time point in the order of the first table structure metadata to obtain the first long string of each time point; perform hash calculation and sum based on the first long string of each time point to obtain the first verification value; obtain the time points contained in each second data shard in the second data shard set, and splice the first fields corresponding to each time point in the order of the second table structure metadata to obtain the second long string of each time point; perform hash calculation and sum based on the second long string of each time point to obtain the second verification value.

[0080] In one embodiment, the comparison and repair module 502 is also used to: determine whether the user terminal specifies a third field, where the third field is a split column used for data sharding; if so, based on the third field and the first field, perform data sharding on the first data table and the second data table respectively.

[0081] In one embodiment, the comparison and repair module 502 is also used to: obtain all numeric type fields in the first data table and / or the second data table; filter out a fourth field with index information from all numeric type fields, and the fourth field is a split column used for data sharding.

[0082] In one embodiment, the comparison and repair module 502 is also used to: if there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization; if so, obtain the comparison times of the shard comparison result in real time, and determine whether the comparison times are greater than the set retry comparison threshold; if greater, perform data repair on the data shards on the write end based on the data shards on the read end until the inconsistent data shards are consistent; otherwise, re-shard and compare the data on the first data table and the second data table based on the data table comparison strategy until the comparison times are greater than the retry comparison threshold.

[0083] Each module in the aforementioned synchronized data table rapid repair device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0084] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store various types of data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for quickly repairing a synchronized data table is implemented.

[0085] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0086] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: Obtaining a first data table on the read side, a second data table on the write side, and a data table comparison strategy; Based on the data table comparison strategy, data sharding and shard comparison are performed on the first data table and the second data table respectively, to obtain shard comparison results of the first data table and the second data table; If there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization. If not, repair the data on the write-end data shard based on the read-end data shard until the inconsistent data shards are consistent.

[0087] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining first table structure metadata of the first data table and second table structure metadata of the second data table, and performing field mapping based on the first table structure metadata and the second table structure metadata to obtain the first field involved in the comparison in the first data table and the second data table.

[0088] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining all time type fields in the first data table and / or the second data table; filtering out the second field containing index information from all the time type fields, where the second field is a partitioning column used for data sharding; based on the first field and the second field, sharding the first data table and the second data table respectively, and calculating the first check value of each data shard in the first data table and the second check value of each data shard in the second data table; comparing the first check value and the second check value of each corresponding data shard to obtain the sharding comparison result of the first data table and the second data table.

[0089] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the second field, data sharding is performed on the first data table and the second data table respectively to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table; the time point contained in each first data shard in the first data shard set is obtained, and the first field corresponding to each time point is spliced ​​in the order of the first table structure metadata to obtain the first long string of each time point; based on the first long string of each time point, hash calculation and summation are performed to obtain the first verification value; based on the time point contained in each second data shard in the second data shard set, the first field corresponding to each time point is spliced ​​in the order of the second table structure metadata to obtain the second long string of each time point; based on the second long string of each time point, hash calculation and summation are performed to obtain the second verification value.

[0090] In one embodiment, when the processor executes the computer program, it also implements the following steps: determining whether the user terminal specifies a third field, where the third field is a split column for data sharding; if so, performing data sharding on the first data table and the second data table respectively based on the third field and the first field.

[0091] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining all numeric type fields in the first data table and / or the second data table; filtering out a fourth field containing index information from all numeric type fields, wherein the fourth field is a split column used for data sharding.

[0092] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if there are inconsistent data shards in the shard comparison result, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if so, the comparison times of the shard comparison result are obtained in real time, and it is determined whether the comparison times are greater than the set retry comparison threshold; if so, data repair is performed on the data shards on the write end based on the data shards on the read end until the data of the inconsistent data shards are consistent; otherwise, data sharding and shard comparison are performed on the first data table and the second data table respectively based on the data table comparison strategy until the comparison times are greater than the retry comparison threshold.

[0093] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps corresponding to the method for quickly repairing a synchronized data table described in the above embodiments are implemented.

[0094] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the steps corresponding to the method for quickly repairing a synchronized data table as described in the above embodiments.

[0095] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0096] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0097] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for quickly repairing a synchronization data table, characterized in that: The method comprises: Obtaining a first data table on a read end, a second data table on a write end, and a data table comparison strategy; Based on the data table comparison strategy, data sharding and shard comparison are performed on the first data table and the second data table respectively to obtain shard comparison results of the first data table and the second data table; If there are inconsistent data shards in the shard comparison result, determine whether the first data table and the second data table are undergoing incremental synchronization; if not, repair the data shards on the write side based on the data shards on the read side until the data of the inconsistent data shards are consistent.

2. The method for quickly repairing a synchronization data table according to claim 1, characterized in that: The data table comparison strategy includes: The first table structure metadata of the first data table and the second table structure metadata of the second data table are obtained, and field mapping is performed based on the first table structure metadata and the second table structure metadata to obtain the first field involved in the comparison in the first data table and the second data table.

3. The method for quickly repairing a synchronization data table according to claim 2, characterized in that: The data table comparison strategy also includes: Obtain all time type fields in the first data table and / or the second data table; Filtering all the time type fields to obtain a second field having index information, where the second field is a segmentation column for data sharding; Based on the first field and the second field, respectively, the first data table and the second data table are segmented, and a first check value of each data segment in the first data table and a second check value of each data segment in the second data table are calculated; The first check value and the second check value of the data slices corresponding to each other are compared to obtain the slice comparison result of the first data table and the second data table.

4. The method for quickly repairing a synchronization data table according to claim 3, characterized in that: The performing data sharding on the first data table and the second data table respectively based on the first field and the second field, and calculating a first check value for each data shard in the first data table and a second check value for each data shard in the second data table comprises: Based on the second field, respectively shard the first data table and the second data table to obtain a first data shard set corresponding to the first data table and a second data shard set corresponding to the second data table; Obtaining the time points contained in each first data shard in the first data shard set, concatenating the first fields corresponding to each of the time points in the order of the first table structure metadata to obtain the first long string of each time point; performing hash calculation and summing based on the first long string of each time point to obtain the first check value; Obtain the time points contained in each second data shard in the second data shard set, and concatenate the first fields corresponding to each of the time points in the order of the second table structure metadata to obtain the second long string for each time point; perform hash calculation and sum based on the second long string for each time point to obtain the second verification value.

5. The method for quickly repairing a synchronization data table according to claim 3, characterized in that: Before acquiring all time type fields in the first data table and / or the second data table, the method further includes: It is determined whether the user terminal specifies a third field, where the third field is a segmentation column for data segmentation; if so, data segmentation is performed on the first data table and the second data table respectively based on the third field and the first field.

6. The method for quickly repairing a synchronization data table according to claim 2, characterized in that: The data table comparison strategy also includes: Acquire all numeric type fields in the first data table and / or the second data table; A fourth field having index information is obtained by screening from all the digital type fields, and the fourth field is a segmentation column used for data sharding.

7. The method for quickly repairing a synchronization data table according to claim 1, characterized in that: The method further comprises: If there are inconsistent data shards in the shard comparison result, determining whether the first data table and the second data table are undergoing incremental synchronization; If so, the comparison times of the shard comparison results are obtained in real time to determine whether the comparison times are greater than a set retry comparison threshold; If it is greater, data repair is performed on the data shards on the write end based on the data shards on the read end until the inconsistent data shards are consistent; otherwise, data sharding and shard comparison are performed on the first data table and the second data table respectively based on the data table comparison strategy until the number of comparisons is greater than the retry comparison threshold.

8. A rapid repair device for a synchronization data table, characterized in that: The device comprises: A data reading module, used to obtain a first data table on a reading end, a second data table on a writing end, and a data table comparison strategy; A comparison and repair module is used to perform data sharding and shard comparison on the first data table and the second data table respectively based on the data table comparison strategy to obtain the shard comparison results of the first data table and the second data table; if there are inconsistent data shards in the shard comparison results, it is determined whether the first data table and the second data table are undergoing incremental synchronization; if not, data repair is performed on the data shards on the write end based on the data shards on the read end until the data of the inconsistent data shards are consistent.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images