Data synchronization method and device and electronic equipment

Abstract

The embodiment of the invention discloses a data synchronization method and device and electronic equipment, and the method comprises the steps: receiving a synchronization request for table structure data in a source database, determining a first fragmentation key based on the table structure data in the source database, and generating a data distribution histogram based on the first fragmentation key and a preset data fragmentation number, based on field selectivity information and field type information of the source database and data distribution information corresponding to the data distribution histogram, determining a second fragmentation key used for fragmenting the source database from the first fragmentation keys, and adjusting the first fragmentation data in the data distribution histogram to obtain second fragmentation data, and on the basis of the value range corresponding to the second fragmentation key and the second fragmentation data, constructing a fragmentation rule used for carrying out fragmentation data extraction processing, extracting the fragmentation data from the source database on the basis of the fragmentation rule, and synchronizing the extracted fragmentation data to the target database. According to the invention, the efficiency and accuracy of data synchronization can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device and electronic equipment. Background Art

[0002] With the growth of data services and the differentiation of different data under different functions and scenarios, the demand for data synchronization between different types of databases is increasing.

[0003] In the related art, the method for synchronizing data between different databases is usually to manually extract data tables by sharding, that is, manually select the primary key or common condition column as the sharding key, and shard the data based on the sharding key, and then synchronize the data based on the sharding. However, the data synchronization processing method of manually sharding data may cause the amount of data on some shards to be too large, and the amount of data on other shards to be too small, resulting in data skew, which in turn affects the efficiency and accuracy of data synchronization. Therefore, it is necessary to provide a data synchronization method that can improve the efficiency and accuracy of data synchronization. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data synchronization method, device and electronic device to solve the problems of poor data synchronization accuracy and low data synchronization efficiency in the prior art.

[0005] To solve the above technical problems, the embodiments of the present invention are implemented as follows: In a first aspect, an embodiment of the present invention provides a data synchronization method, the method comprising: Receive a synchronization request for table structure data in a source database; Determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; Based on the field selectivity information and field type information of the source database and the data distribution information corresponding to the data distribution histogram, determine a second sharding key for sharding the source database from the first sharding key; Based on the data volume corresponding to each first shard data in the data distribution histogram and the average data volume of the shards, the first shard data in the data distribution histogram is adjusted so that the data volume difference between each shard data is less than a preset data volume threshold, to obtain second shard data; Based on the value range corresponding to the second sharding key and the second sharding data, a sharding rule for performing sharding extraction processing is constructed, sharding data is extracted from the source database based on the sharding rule, and the extracted sharding data is synchronized to the target database.

[0006] In a second aspect, an embodiment of the present invention provides a data synchronization device, the device comprising: A request receiving module, used for receiving a synchronization request for table structure data in a source database; A data distribution histogram generation module, configured to determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; a second sharding key determination module, configured to determine a second sharding key for sharding the source database from the first sharding key based on the field selectivity information and field type information of the source database and the data distribution information corresponding to the data distribution histogram; A data adjustment module, configured to adjust the first shard data in the data distribution histogram based on the data volume corresponding to each first shard data and the average data volume of the shards so that the data volume difference between each shard data is less than a preset data volume threshold, thereby obtaining second shard data; A data synchronization module is used to construct a sharding rule for sharding extraction processing based on the value range corresponding to the second sharding key and the second sharding data, extract sharding data from the source database based on the sharding rule, and synchronize the extracted sharding data to the target database.

[0007] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data synchronization method provided in the above embodiment.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method provided in the above embodiment are implemented.

[0009] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the data synchronization method provided in the above embodiment are implemented.

[0010] It can be seen from the technical solution provided by the above embodiments of the present invention that the embodiments of the present invention, by receiving a synchronization request for table structure data in a source database, first determines a first sharding key based on the table structure data in the source database, and generates a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards. Secondly, based on the field selectivity information, field type information of the source database and the data distribution information corresponding to the data distribution histogram, a second sharding key for sharding the source database is determined from the first sharding key. Then, based on the data volume corresponding to each first sharding data in the data distribution histogram and the average data volume of the shards, the first sharding data in the data distribution histogram is adjusted so that the data volume difference between each sharding data is less than a preset data volume threshold, and the second sharding data is obtained. Finally, based on the value range corresponding to the second sharding key and the second sharding data, a sharding rule for sharding extraction processing is constructed, sharding data is extracted from the source database based on the sharding rule, and the extracted sharding data is synchronized to the target database. In this way, by collecting the table structure data in the source database and generating a data distribution histogram, the first sharding key and the first sharding data are preliminarily determined, and the second sharding key is determined based on the field selectivity information, field type information, and data distribution information corresponding to the data distribution histogram, which can improve the accuracy of sharding key selection, thereby improving the accuracy and efficiency of subsequent data synchronization. By adjusting the first sharding data in the data distribution histogram to obtain the second sharding data, data skew can be effectively avoided, and the uniformity and balance of data distribution can be improved, thereby improving the efficiency of data concurrent processing, thereby improving the efficiency of data synchronization. Determining the second sharding key based on the field selectivity information, field type information, and data distribution information corresponding to the data distribution histogram can objectively reflect the selection performance, query performance, and data distribution uniformity of the sharding key, thereby improving the reliability and accuracy of the second sharding key, thereby improving the accuracy and efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 A schematic diagram of a data synchronization method according to the present invention; Figure 2 A schematic diagram of a data distribution histogram of the present invention; Figure 3 A schematic diagram of the principle of a data synchronization method of the present invention; Figure 4It is a structural schematic diagram of a data synchronization device of the present invention; Figure 5 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0013] Embodiments of the present invention provide a data synchronization method, device and electronic device.

[0014] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0015] like Figure 1 As shown, an embodiment of this specification provides a data synchronization method, and the execution subject of the method can be a terminal device or a server, wherein the terminal device can be a certain terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car device, etc.). The server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server in the financial field or the online shopping field, or a background server of an application, etc. In this embodiment, the server is used as an example for detailed description. For the execution process of the terminal device, please refer to the following related content, which will not be repeated here. The method can specifically include the following steps: In step S102 , a synchronization request for table structure data in a source database is received.

[0016] In the field of data processing, with the growth of data services, the segmentation of functions and scenarios, and the increasing demand for database updates, the demand for data synchronization between different types of databases is also increasing. One of the key issues in data synchronization is the data synchronization of large tables, which are data tables containing a large number of data rows in the database. The embodiments of this specification synchronize the full amount of data in the large table of the source database to the target database by performing data sharding on the source database.

[0017] The table structure data includes: one or more of data field types, indexes, and constraints.

[0018] In step S104, a first sharding key is determined based on the table structure data in the source database, and a data distribution histogram corresponding to the source database is generated based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key.

[0019] In implementation, the first sharding key is determined based on the table structure data in the source database. The table structure data may be directly parsed to obtain the first sharding key, or the first sharding key may be determined after priority selection of the sharding keys.

[0020] The method of generating a data distribution histogram corresponding to the source database based on the first sharding key and the preset number of data shards can determine the value range of the sharding field in the source database (i.e., multiple first sharding data) based on the first sharding key, and then determine the group distance of the data distribution histogram based on the value range and the preset data sharding data, and finally generate the data distribution histogram corresponding to the source database according to the group distance and the value range. The horizontal axis of the data distribution histogram is the value range of the field corresponding to the first sharding key, and the vertical axis is the amount of data corresponding to the value range (i.e., the total number of table records whose values ​​fall within the value range). The schematic diagram of the data distribution histogram in the embodiment of this specification can be found in Figure 2 shown.

[0021] In step S106, based on the field selectivity information, field type information of the source database, and data distribution information corresponding to the data distribution histogram, a second sharding key for sharding the source database is determined from the first sharding key.

[0022] The field selectivity information of the first shard key in the source database refers to the information that after determining field A as the first shard key, based on the jitter of the cardinality that the first shard key can return, if only one record is returned when a value field A of the first shard key is entered, then field A is judged to have better field selectivity, and if multiple records are returned when a value field B of the first shard key is entered, then field B is judged to have poor field selectivity. For example: for a database with ID card statistics, entering the ID card number (field A) can determine a unique person, and entering the name (field B) can get multiple people. Based on this, it is judged that the field selectivity of the ID card number is better than that of the name.

[0023] The field type information of the first shard key in the source database is used to measure the query performance of the first shard key as an index. If a value field C of the first shard key is entered and only one record is returned, the query performance of field C as an index is better. If a value field D of the first shard key is entered and multiple records are returned, the query performance of field D as an index is worse.

[0024] The data distribution information of the first shard key in the source database refers to the uniformity of data distribution.

[0025] In implementation, based on the field selectivity information, field type information and data distribution information corresponding to the data distribution histogram of the source database, the method of determining the second sharding key for sharding the source database from the first sharding key can be to directly select the second sharding key from the first sharding key based on the field selectivity information, field type information and data distribution information corresponding to the data distribution histogram, or to sort the first sharding keys in descending order of weight values ​​based on the field selectivity information, field type information and weight values ​​of the data distribution information corresponding to the data distribution histogram, and determine the second sharding key based on the first sharding key with a higher weight value.

[0026] In step S108, based on the data volume corresponding to each first shard data in the data distribution histogram and the average data volume of the shards, the first shard data in the data distribution histogram is adjusted so that the data volume difference between each shard data is less than a preset data volume threshold, thereby obtaining second shard data.

[0027] In implementation, based on the data volume corresponding to each first shard data in the data distribution histogram and the average data volume of the shards, the first shard data in the data distribution histogram can be adjusted by sorting the data volume corresponding to the first shard data in descending order, merging the data volume that is ranked later, thereby reducing the data volume difference between each shard. It is also possible to merge shards with smaller data volumes and split shards with larger data volumes based on the average data volume of the shards, thereby reducing the data volume difference between each shard and improving the concurrent efficiency of data synchronization.

[0028] In step S110, based on the value range corresponding to the second shard key and the second shard data, a shard rule for performing shard extraction processing is constructed, shard data is extracted from the source database based on the shard rule, and the extracted shard data is synchronized to the target database.

[0029] Compared with the first sharding key, the second sharding key determined by step S106 can further improve the accuracy of data sharding, which is beneficial to improving the efficiency of data synchronization. The second sharding data obtained by adjusting the first sharding data in the data distribution histogram in step S108 can effectively reduce data skew and rebalance the skewed data, which is beneficial to improving the uniformity of data distribution and the concurrency efficiency of subsequent data sharding extraction. After determining the second sharding data, the value range corresponding to the second sharding data can be determined, and the sharding rules for sharding extraction processing can be constructed based on the value range corresponding to the second sharding key and the second sharding data in step S110.

[0030] In implementation, based on the value range corresponding to the second sharding key and the second sharding data, a method for constructing a sharding rule for sharding number extraction processing can be based on the value range corresponding to the second sharding key and the second sharding data, and in the order of the weight value of the second sharding key from high to low, generate a sharding rule, and the sharding rule can be implemented in the form of a recommendation list, that is: in the order of the weight value of the second sharding key from high to low, generate a recommendation list corresponding to one or more second sharding key fields, so that the user or system administrator can select the corresponding recommendation list according to the current application scenario. The recommendation list may include the field name of the second sharding key, the weight value, the second sharding data, and the value range corresponding to the second sharding data.

[0031] In the embodiment of this specification, the process of determining the first sharding key based on the table structure data in the source database in step S104 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1042-S1044.

[0032] In step S1042, the table structure data in the source database is parsed to obtain a plurality of pre-selected partition keys.

[0033] In step S1044, based on a preset priority order, a plurality of pre-selected partition keys are sorted to obtain a first shard key, where the preset priority order is a priority order related to the table structure data.

[0034] In a possible implementation, the preset priority order can be set as: "numeric primary key" > "numeric unique index" > "time primary key" > "time unique index" > "other primary keys" > "other unique indexes" > "common indexes". That is, when determining the first sharding key, the numeric primary key and unique index are considered first, then the time primary key and unique index are considered, and finally the other primary keys, unique indexes and common indexes are considered.

[0035] Furthermore, before step S1042, step S1040 may be included: obtaining table structure data of the source database.

[0036] In implementation, the table structure data of the source database can be obtained by querying the system tables and / or system views used to store metadata in the source database through the database connection interface. For example, the source database can be connected through JDBC (Java Database Connectivity, a standard application program interface used in Java to connect to and operate databases), the table structure data in the source database can be queried and cached for subsequent parsing and processing.

[0037] Furthermore, after step S1044, step S1046 may be further included: determining whether there is a first partition key with a priority order; Step S1048: If yes, calculate the field cardinality of the first partition key that is ranked in priority order, and determine a unique first partition key based on the field cardinality.

[0038] Through the above steps S1046 and S1048, a unique first partition key can be determined, thereby improving the accuracy of partition key determination.

[0039] In the embodiment of this specification, the processing of generating a data distribution histogram corresponding to the source database based on the first sharding key and the preset number of data shards in step S104 can be varied. An optional processing method is provided below, and specific details can be referred to the processing of the following steps B2-B6.

[0040] In step B2, the value range of the shard field in the source database and the preset number of data shards are determined based on the first partition key.

[0041] The value range of the shard field can be understood as the maximum and minimum values ​​of the shard field.

[0042] In step B4, based on the determined value range of the sharding field and the preset number of data shards, the class distance of the data distribution histogram is determined.

[0043] You can use the formula , calculate the group distance of the data distribution histogram, where, is the group distance, is the maximum value of the fragment field, is the minimum value of the fragment field, The preset number of data shards.

[0044] In step B6, a data distribution histogram corresponding to the source database is generated according to the determined group interval and the value range of the sharding field.

[0045] In the embodiment of this specification, the processing of the above step S106 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1062-S1066.

[0046] In step S1062, based on the field selectivity information, field type information, and data distribution information of the source database, the field selectivity coefficient, field type coefficient, and distribution uniformity coefficient corresponding to each first sharding key are determined respectively.

[0047] By weighing the impact of multiple factors such as field selectivity information, field type information, and data distribution information on the sharding effect to select the second sharding key, the reliability and accuracy of the determined second sharding key can be improved.

[0048] In implementation, the field selectivity coefficient, field type coefficient, and distribution uniformity coefficient corresponding to the first sharding key can be seen in the following Table 1.

[0049]

[0050] Table 1 In step S1064, the product of the field selectivity coefficient, the field type coefficient, and the distribution uniformity coefficient corresponding to each first sharding key is calculated to obtain the weight value of each first sharding key, and the second sharding key is determined from the first sharding key based on the weight value.

[0051] By calculating the product of the field selectivity coefficient, field type coefficient, and distribution uniformity coefficient corresponding to each first sharding key, the weight value of each first sharding key is obtained, and then the weight values ​​are sorted in descending order. Based on the current application scenario, the first sharding key with the top N weight values ​​is selected and determined as the second sharding key.

[0052] In the embodiments of this specification, the processing of the above step S108 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1080-S1088.

[0053] In step S1080, based on the data volume corresponding to each first shard data in the data distribution histogram and the preset number of data shards, the average shard data volume is calculated.

[0054] The average data volume of the shards is also called the average grouping frequency, the preset number of data shards is also called the number of groups, and the data volume corresponding to each first shard data is also called the frequency. The way to calculate the average data volume of the shards can be: the average data volume of the shards = the sum of the data volume corresponding to each first shard data / the preset number of data shards.

[0055] In step S1082, first shard data whose data volume is smaller than the average data volume of the shards is selected from the data distribution histogram, and the selected first shard data is merged to obtain third shard data. The number of first shard data to be merged is the minimum number that makes the data volume corresponding to the third shard data larger than the product of the average data volume of the shards and the first threshold value, and the first threshold value is less than 1.

[0056] In implementation, the selected first shard data is merged to obtain the third shard data by random merging or sequential merging in order of size. The first threshold can be set according to the application scenario, for example, 0.8, 0.9, etc.

[0057] In a possible implementation, the first shard data with a data volume smaller than the average data volume of the shard can be first selected from the data distribution histogram, and then the first shard data with a data volume smaller than the average data volume of the shard can be sorted in order of frequency from small to large, and then the data distribution histogram coordinates are used as a reference, starting from the smallest frequency and merging from left to right. If the frequency after merging is smaller than the product of the average data volume of the shard and the first threshold, the merging continues to the right until the frequency after merging is greater than or equal to the product of the average data volume of the shard and the first threshold. The data finally obtained by merging is the third shard data. After obtaining the third shard data, the value range corresponding to the third shard data is recorded. See Figure 2 Taking the first threshold value of 0.8 as an example, as the value range of a single slice, the value range corresponding to the third slice data can be expressed as: [10, 15] + [40, 45] + [200, 205], where [10, 15], [40, 45], and [200,205] are the horizontal coordinates of the three first slice data whose data volume is less than the average data volume of the slice. After the above three first slice data are merged, a second slice data is obtained.

[0058] In step S1084, first shard data whose data volume is greater than or equal to the product of the average data volume of the shard and the second threshold is selected from the data distribution histogram, and the selected first shard data is split based on the group distance to obtain fourth shard data. The group distance is determined according to the value range of the shard data and the preset number of data shards.

[0059] In a possible implementation, the second threshold value is 2, that is, the first shard data having a data volume greater than or equal to twice the average data volume of the shard is selected from the data distribution histogram.

[0060] In implementation, taking the second threshold value of 2 as an example, the first shard data whose data volume is greater than or equal to 2 times the average data volume of the shard can be first selected from the data distribution histogram, and then the first shard data is split according to the multiple (rounded) of the average number, and the splitting method is to reduce the group interval to generate sub-groups, and the group interval of the sub-group = the group interval of the parent group / the multiple compared with the average number of shards. Each sub-group is used as a fourth shard data. After obtaining the fourth shard data, the value range corresponding to the fourth shard data is recorded.

[0061] In step S1086, first shard data other than the third shard data and the fourth shard data is selected from the data distribution histogram, and fifth shard data is determined based on the selected first shard data.

[0062] In implementation, each first shard data other than the third shard data and the fourth shard data in the data distribution histogram may be taken as a fifth shard data, and the value range corresponding to the fifth shard data may be recorded.

[0063] In step S1088, a set of the third fragment data, the fourth fragment data, and the fifth fragment data is used as the second fragment data.

[0064] Through the above steps S1080-S1088, the first shard data in the data distribution histogram is adjusted, low-frequency groups are merged (i.e., the first shard data whose merged data volume is less than the average data volume of the shard), and high-frequency groups are split (i.e., the first shard data whose data volume is greater than or equal to the product of the average data volume of the shard and the second threshold) The obtained second shard data can further effectively avoid data skew and improve the uniformity of data distribution. When the shard data is subsequently extracted from the source database, the data concurrency efficiency can be further effectively improved, thereby improving the data synchronization efficiency.

[0065] In the embodiment of this specification, the process of extracting shard data from the source database based on the sharding rule in the above step S110 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps C2-C4.

[0066] In step C2, a pool connection is performed on the source database and the target database.

[0067] By establishing a pooled connection between the source database and the target database, the number of visits to the target database can be reduced, the pressure on the target database can be reduced, the system performance can be improved, and the database responsiveness can be improved. By reusing the pre-created database connection, when the application needs to connect to the corresponding database, it can be directly obtained and used, which is conducive to shortening the system response time and improving the responsiveness. In addition, by establishing a pooled connection between the source database and the target database, it is also conducive to controlling resource usage. The connection pool of the pooled connection can control the number of database connections, avoid resource waste and high load exceptions caused by frequent connection operations, and thus enhance system stability.

[0068] In step C4, based on the second sharding key in the sharding rule and the value range corresponding to the second sharding data, multiple second sharding data are concurrently extracted from the source database, and each second sharding data corresponds to a thread.

[0069] By concurrently extracting multiple second shard data (i.e., initiating the extraction of multiple data shards at the same time) and allocating a separate thread to each second shard data, it is helpful to improve the throughput of data extraction and improve data synchronization efficiency.

[0070] Furthermore, the process of extracting shard data from the source database based on the sharding rule in the above step S110 may also include step C6: when extracting multiple second shard data concurrently from the source database, performing read performance monitoring on the extraction process.

[0071] In implementation, a global data area can be allocated in the source database to count the extraction performance index data of the thread corresponding to each second shard data, and the extraction performance index data mainly includes one or more of the extraction rate per second, the number of extracted data bytes per second, and the CPU / memory usage. This enables timely acquisition of the execution efficiency and health of the concurrent extraction process, which is conducive to improving the stability and reliability of data synchronization.

[0072] In the embodiment of the present specification, the process of synchronizing the extracted shard data to the target database in the above step S110 can be varied, and four optional processing methods are provided below.

[0073] In the first processing method, field compatibility conversion is performed on the extracted shard data to obtain the processed shard data, and the processed shard data is synchronized to the target database.

[0074] During implementation, the metadata information (such as table structure information, data distribution information, etc.) in the source database and the target database can be compared, and the source data can be converted in precision and / or data format according to preset data conversion rules. Then, the converted sharded data can be synchronized to the target database. This method can ensure that the data in the source database can be smoothly written into the target database, which is conducive to improving the efficiency of data synchronization.

[0075] In the second processing method, based on the type of the target database, the structured query language corresponding to the extracted shard data is processed for storage and assembly, and the extracted shard data after the storage and assembly process is synchronized to the target database.

[0076] In implementation, the incoming structured query language that complies with preset grammatical rules can be assembled according to the type of the target database, and the assembled incoming structured query language can be grammatically optimized to improve the processing efficiency of the structured query language and further improve the data synchronization efficiency.

[0077] In the third processing method, the extracted shard data is synchronized to the target database by batch merging and executing the structured query language corresponding to the extracted shard data.

[0078] In implementation, multiple SQL (Structured Query Language) statements can be combined into a statement block, and the statement variables are assigned values ​​through binding variables, and then the statement block is executed. Compared with the single SQL execution method, the batch merging execution processing method can significantly reduce the number of database requests and network IO overhead, which is conducive to improving execution efficiency.

[0079] The fourth processing method is to combine the above three processing methods to further improve data synchronization efficiency.

[0080] Furthermore, the process of extracting shard data from the source database based on the sharding rule in the above step S110 may also include the step of: when synchronizing the extracted shard data to the target database, monitoring the write performance of the data synchronization process.

[0081] During implementation, the rate at which data is entered into the target database can be counted, thereby monitoring the impact of the data synchronization process on the performance of the target database, which is beneficial to improving the stability and reliability of data synchronization.

[0082] The schematic diagram of the principle of data synchronization in the embodiments of this specification can be found in Figure 3 , Figure 3 In the process of metadata collection, the table structure data is collected. Figure 3 It can be seen that after obtaining the table structure information and data distribution of the source database through metadata collection and statistical information collection, the sharding rules are calculated by calculating the partition key (the second partition key) and the partition range (the value range corresponding to the second sharding data). Then, based on the sharding rules, a sharding structured query language is constructed to generate sharding extraction SQL, data shards are extracted from the source database, and the extracted data is written into the target database, thereby synchronizing the data from the source database to the target database.

[0083] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0084] The above is a data synchronization method provided by one or more embodiments of this specification. Based on the same idea, one or more embodiments of this specification also provide a data synchronization device, such as Figure 4 shown.

[0085] The data synchronization device includes: a request receiving module 210, a data distribution histogram generating module 220, a second sharding key determining module 230, a data adjustment module 240 and a data synchronization module 250. The request receiving module 210 is used to receive a synchronization request for table structure data in a source database; A data distribution histogram generation module 220 is used to determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; A second sharding key determination module 230, configured to determine a second sharding key for sharding the source database from the first sharding key based on the field selectivity information, the field type information, and the data distribution information corresponding to the data distribution histogram of the source database; The data adjustment module 240 is used to adjust the first shard data in the data distribution histogram based on the data volume corresponding to each first shard data and the average data volume of the shards so that the data volume difference between each shard data is less than a preset data volume threshold, thereby obtaining second shard data; The data synchronization module 250 is used to construct a sharding rule for sharding extraction processing based on the value range corresponding to the second sharding key and the second sharding data, extract sharding data from the source database based on the sharding rule, and synchronize the extracted sharding data to the target database.

[0086] In the embodiment of this specification, the data distribution histogram generation module 220 includes: A data parsing and processing unit parses and processes the table structure data in the source database to obtain multiple pre-selected partition keys; The sorting unit sorts the multiple pre-selected partition keys based on a preset priority order to obtain a first partition key, where the preset priority order is a priority order related to the table structure data.

[0087] The data distribution histogram generating unit generates a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key.

[0088] In the embodiment of this specification, the second shard key determination module 230 includes: A coefficient determination unit, based on the field selectivity information, field type information and data distribution information of the source database, determines the field selectivity coefficient, field type coefficient and distribution uniformity coefficient corresponding to each first sharding key respectively; The weight calculation unit calculates the product of the field selectivity coefficient, the field type coefficient, and the distribution uniformity coefficient corresponding to each first sharding key to obtain the weight value of each first sharding key, and determines the second sharding key from the first sharding key based on the weight value.

[0089] In the embodiment of this specification, the data adjustment module 240 includes: A shard average data volume calculation unit, which calculates the shard average data volume based on the data volume corresponding to each first shard data in the data distribution histogram and the preset number of data shards; a merging unit, which selects first shard data whose data volume is smaller than the average data volume of the shards from the data distribution histogram, and performs merging processing on the selected first shard data to obtain third shard data, wherein the number of the first shard data to be merged is the minimum number that makes the data volume corresponding to the third shard data larger than the product of the average data volume of the shards and a first threshold value, and the first threshold value is smaller than 1; a splitting unit, selecting first shard data whose data volume is greater than or equal to the product of the average data volume of the shards and a second threshold from the data distribution histogram, and performing splitting processing on the selected first shard data based on a group distance to obtain fourth shard data, wherein the group distance is determined according to a value range of the shard data and a preset number of data shards, and the second threshold is greater than 1; a fifth shard data determining unit, which selects first shard data other than the third shard data and the fourth shard data from the data distribution histogram, and determines fifth shard data based on the selected first shard data; The second fragment data determining unit uses the set of the third fragment data, the fourth fragment data and the fifth fragment data as the second fragment data.

[0090] In the embodiment of this specification, the data synchronization module 250 includes: A sharding rule construction unit constructs a sharding rule for performing sharding number extraction processing based on a value range corresponding to the second sharding key and the second sharding data; A pooled connection unit, which performs pooled connections on the source database and the target database; A concurrent extraction unit concurrently extracts a plurality of second shard data from a source database based on a second shard key in a shard rule and a value range corresponding to the second shard data, wherein each second shard data corresponds to a thread; The first synchronization unit performs field compatibility conversion processing on the extracted shard data, obtains the processed shard data, and synchronizes the processed shard data to the target database; The second synchronization unit performs storage assembly processing on the structured query language corresponding to the extracted shard data based on the type of the target database, and synchronizes the extracted shard data after the storage assembly processing to the target database; The third synchronization unit synchronizes the extracted shard data to the target database by batch merging and executing the structured query language corresponding to the extracted shard data.

[0091] Those skilled in the art should understand that the above-mentioned data synchronization device can be used to implement the data synchronization method described above, and the detailed description thereof should be similar to the description of the method part above, and will not be further described here to avoid tediousness.

[0092] Based on the same idea, one or more embodiments of this specification also provide an electronic device, such as Figure 5 The electronic device 300 includes but is not limited to: a radio frequency unit 301, a network module 302, an audio output unit 303, an input unit 304, a sensor 305, a display unit 306, a user input unit 307, an interface unit 308, a memory 309, a processor 310, and a power supply 311. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0093] The processor 310 is used to: receive a synchronization request for table structure data in a source database; Determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; Based on the field selectivity information and field type information of the source database and the data distribution information corresponding to the data distribution histogram, determine a second sharding key for sharding the source database from the first sharding key; Based on the data volume corresponding to each first shard data in the data distribution histogram and the average data volume of the shards, the first shard data in the data distribution histogram is adjusted so that the data volume difference between each shard data is less than a preset data volume threshold, to obtain second shard data; Based on the value range corresponding to the second sharding key and the second sharding data, a sharding rule for performing sharding extraction processing is constructed, sharding data is extracted from the source database based on the sharding rule, and the extracted sharding data is synchronized to the target database.

[0094] The processor 310 is further configured to: parse the table structure data in the source database to obtain a plurality of pre-selected partition keys; Based on a preset priority order, the multiple pre-selected partition keys are sorted to obtain a first shard key, and the preset priority order is a priority order related to the table structure data.

[0095] The processor 310 is further configured to: determine the field selectivity coefficient, field type coefficient, and distribution uniformity coefficient corresponding to each first sharding key based on the field selectivity information, field type information, and data distribution information of the source database; The product of the field selectivity coefficient, the field type coefficient, and the distribution uniformity coefficient corresponding to each first sharding key is calculated to obtain a weight value of each first sharding key, and a second sharding key is determined from the first sharding keys based on the weight value.

[0096] The processor 310 is further configured to: calculate an average data volume of each shard based on the data volume corresponding to each first shard data in the data distribution histogram and the preset number of data shards; Selecting first shard data whose data volume is smaller than the average data volume of the shards from the data distribution histogram, and merging the selected first shard data to obtain third shard data, wherein the number of first shard data to be merged is the minimum number that makes the data volume corresponding to the third shard data larger than the product of the average data volume of the shards and a first threshold value, and the first threshold value is smaller than 1; Selecting first shard data whose data volume is greater than or equal to the product of the average data volume of the shards and a second threshold from the data distribution histogram, and splitting the selected first shard data based on the group distance to obtain fourth shard data, wherein the group distance is determined according to a value range of the shard data and a preset number of data shards, and the second threshold is greater than 1; Selecting first shard data other than the third shard data and the fourth shard data from the data distribution histogram, and determining fifth shard data based on the selected first shard data; The set of the third fragment data, the fourth fragment data and the fifth fragment data is used as the second fragment data.

[0097] In addition, the processor 310 is further configured to: perform pooled connection on the source database and the target database; Based on the second sharding key in the sharding rule and the value range corresponding to the second sharding data, multiple second sharding data are concurrently extracted from the source database, and each second sharding data corresponds to a thread.

[0098] In addition, the processor 310 is further configured to: perform field compatible conversion processing on the extracted shard data, obtain the processed shard data, and synchronize the processed shard data to the target database; or, Based on the type of the target database, the structured query language corresponding to the extracted shard data is put into storage and assembled, and the extracted shard data after the storage and assembly process is synchronized to the target database; or, The extracted shard data is synchronized to the target database by batch merging and executing the structured query language corresponding to the extracted shard data.

[0099] It should be understood that in the embodiment of the present invention, the radio frequency unit 301 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 310 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 301 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 301 can also communicate with the network and other electronic devices through a wireless communication system.

[0100] The electronic device provides users with wireless broadband Internet access through the network module 302, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0101] The audio output unit 303 can convert the audio data received by the RF unit 301 or the network module 302 or stored in the memory 309 into an audio signal and output it as sound. Moreover, the audio output unit 303 can also provide audio output related to a specific function performed by the electronic device 300 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 303 includes a speaker, a buzzer, a receiver, etc.

[0102] The input unit 304 is used to receive audio or video signals. The input unit 304 may include a graphics processor (GPU) 3041 and a microphone 3042. The graphics processor 3041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 306. The image frame processed by the graphics processor 3041 can be stored in the memory 309 (or other storage medium) or sent via the radio frequency unit 301 or the network module 302. The microphone 3042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 301 in the case of a telephone call mode.

[0103] The electronic device 300 also includes at least one sensor 305, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 3061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 3061 and / or the backlight when the electronic device 300 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 305 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0104] The display unit 306 is used to display information input by the user or information provided to the user. The display unit 306 may include a display panel 3061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0105] The user input unit 307 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 307 includes a touch panel 3031 and other input devices 3072. The touch panel 3031, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 3031). The touch panel 3031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 310, receives the command sent by the processor 310 and executes it. In addition, the touch panel 3031 can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 3031, the user input unit 307 may also include other input devices 3072. Specifically, other input devices 3072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0106] Furthermore, the touch panel 3031 may be overlaid on the display panel 3061. When the touch panel 3031 detects a touch operation on or near it, it is transmitted to the processor 310 to determine the type of touch event. Then, the processor 310 provides corresponding visual output on the display panel 3061 according to the type of touch event. Figure 5 In the figure, the touch panel 3031 and the display panel 3061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 3031 and the display panel 3061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0107] The interface unit 308 is an interface for connecting an external device to the electronic device 300. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 308 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 300 or may be used to transmit data between the electronic device 300 and an external device.

[0108] The memory 309 can be used to store software programs and various data. The memory 309 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 309 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 309 and calling data stored in the memory 309, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 310 may include one or more processing units; preferably, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0110] The electronic device 300 may also include a power source 311 (such as a battery) for supplying power to various components. Preferably, the power source 311 may be logically connected to the processor 310 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0111] Preferably, an embodiment of the present invention further provides an electronic device, comprising a processor 310, a memory 309, and a computer program stored in the memory 309 and executable on the processor 310. When the computer program is executed by the processor 310, each process of the above-mentioned data synchronization method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0112] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data synchronization method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0113] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the various processes of the above-mentioned data synchronization method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0114] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0120] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0122] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data synchronization method, characterized in that: The method comprises: Receive a synchronization request for table structure data in a source database; Determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; Based on the field selectivity information and field type information of the source database and the data distribution information corresponding to the data distribution histogram, determine a second sharding key for sharding the source database from the first sharding key; Based on the data volume corresponding to each first shard data in the data distribution histogram and the average data volume of the shards, the first shard data in the data distribution histogram is adjusted so that the data volume difference between each shard data is less than a preset data volume threshold, to obtain second shard data; Based on the value range corresponding to the second sharding key and the second sharding data, a sharding rule for performing sharding extraction processing is constructed, sharding data is extracted from the source database based on the sharding rule, and the extracted sharding data is synchronized to the target database.

2. The method according to claim 1, characterized in that The table structure data includes: one or more of a field type, an index, and a constraint of the data. The determining of the first sharding key based on the table structure data in the source database includes: Parsing the table structure data in the source database to obtain multiple pre-selected partition keys; Based on a preset priority order, the multiple pre-selected partition keys are sorted to obtain a first shard key, and the preset priority order is a priority order related to the table structure data.

3. The method according to claim 1, characterized in that The determining, from the first sharding key, a second sharding key for sharding the source database based on the field selectivity information and the field type information of the source database and the data distribution information corresponding to the data distribution histogram, includes: Based on the field selectivity information, field type information, and data distribution information of the source database, determine the field selectivity coefficient, field type coefficient, and distribution uniformity coefficient corresponding to each first sharding key respectively; The product of the field selectivity coefficient, the field type coefficient, and the distribution uniformity coefficient corresponding to each first sharding key is calculated to obtain a weight value of each first sharding key, and a second sharding key is determined from the first sharding keys based on the weight value.

4. The method according to claim 1, characterized in that: The method of adjusting the first shard data in the data distribution histogram based on the data volume corresponding to each first shard data and the average data volume of the shards so that the data volume difference between each shard data is less than a preset data volume threshold to obtain the second shard data includes: Based on the data volume corresponding to each first shard data in the data distribution histogram and the preset number of data shards, the average data volume of the shards is calculated; Selecting first shard data whose data volume is smaller than the average data volume of the shards from the data distribution histogram, and merging the selected first shard data to obtain third shard data, wherein the number of first shard data to be merged is the minimum number that makes the data volume corresponding to the third shard data larger than the product of the average data volume of the shards and a first threshold value, and the first threshold value is smaller than 1; Selecting first shard data whose data volume is greater than or equal to the product of the average data volume of the shards and a second threshold from the data distribution histogram, and splitting the selected first shard data based on the group distance to obtain fourth shard data, wherein the group distance is determined according to a value range of the shard data and a preset number of data shards, and the second threshold is greater than 1; Selecting first shard data other than the third shard data and the fourth shard data from the data distribution histogram, and determining fifth shard data based on the selected first shard data; The set of the third fragment data, the fourth fragment data and the fifth fragment data is used as the second fragment data.

5. The method according to claim 1, characterized in that The extracting the shard data from the source database based on the shard rule includes: Performing pooled connection on the source database and the target database; Based on the second sharding key in the sharding rule and the value range corresponding to the second sharding data, multiple second sharding data are concurrently extracted from the source database, and each second sharding data corresponds to a thread.

6. The method according to claim 1, characterized in that Synchronize the extracted shard data to the target database, including: Performing field-compatible conversion processing on the extracted shard data, obtaining the processed shard data, and synchronizing the processed shard data to the target database; or, Based on the type of the target database, the structured query language corresponding to the extracted shard data is put into storage and assembled, and the extracted shard data after the storage and assembly process is synchronized to the target database; or, The extracted shard data is synchronized to the target database by batch merging and executing the structured query language corresponding to the extracted shard data.

7. A data synchronization device, characterized in that: The device comprises: A request receiving module, used for receiving a synchronization request for table structure data in a source database; A data distribution histogram generation module, configured to determine a first sharding key based on the table structure data in the source database, and generate a data distribution histogram corresponding to the source database based on the first sharding key and a preset number of data shards, wherein the data distribution histogram includes a plurality of first sharding data divided based on the first sharding key; a second sharding key determination module, configured to determine a second sharding key for sharding the source database from the first sharding key based on the field selectivity information and field type information of the source database and the data distribution information corresponding to the data distribution histogram; A data adjustment module, configured to adjust the first shard data in the data distribution histogram based on the data volume corresponding to each first shard data and the average data volume of the shards so that the data volume difference between each shard data is less than a preset data volume threshold, thereby obtaining second shard data; A data synchronization module is used to construct a sharding rule for sharding extraction processing based on the value range corresponding to the second sharding key and the second sharding data, extract sharding data from the source database based on the sharding rule, and synchronize the extracted sharding data to the target database.

8. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data synchronization method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 6.

Images