Data synchronization method, device and electronic device based on slicing processing

Abstract

The present invention discloses a data synchronization method, device and electronic device based on slicing processing, the method comprising: configuring a virtual slicing value according to the data type of the primary key of a data table; dividing the primary key of the data table into multiple slicing intervals based on the virtual slicing value; slicing the primary key of each slicing interval to obtain multiple sub-slicing intervals; and synchronizing data according to each sub-slicing interval. The present invention configures a virtual slicing value to distinguish primary keys of different continuous storage ranges, and divides primary keys of the same continuous storage range into the same slicing interval through the virtual slicing value, so that each slicing interval only contains primary keys of the same continuous storage range, ensuring that the primary key value of each slicing interval is continuous and there is no jump, and then slicing the primary key of each slicing interval to ensure that the number of data items contained in each sub-slicing interval in each slicing interval after slicing is the same as much as possible, avoiding data skew, thereby reducing data synchronization time and improving data synchronization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization technology, and in particular to a data synchronization method, device, electronic device and computer-readable medium based on slicing processing. Background Art

[0002] Data synchronization involves synchronizing data from different sources to another using various technologies and tools. In big data processing, this typically involves synchronizing data from relational databases to data warehouses. Currently, relational databases use primary keys with different data types to store large amounts of business data.

[0003] During data synchronization, data is typically split into N (N > 1 and a natural number) parts based on the primary key, and then N tasks are started to synchronize these N parts in parallel. If the primary keys use different data types, the differences in the value ranges of the different data types can lead to significant differences in the number of data items contained in each part after the split. Consequently, the number of data items synchronized by each task also varies significantly, causing data skew and slow data synchronization. Summary of the Invention

[0004] In view of this, the main purpose of the present invention is to propose a data synchronization method, device, electronic device and computer-readable medium based on slicing processing, in order to at least partially solve at least one of the above technical problems.

[0005] In order to solve the above technical problems, the first aspect of the present invention proposes a data synchronization method based on slicing processing, the method comprising:

[0006] Configure the virtual split value according to the data type of the primary key of the data table;

[0007] Dividing the data table primary key into multiple segmentation intervals based on the virtual segmentation value;

[0008] Slice the primary key of each partition interval to obtain multiple sub-partition intervals;

[0009] Data synchronization is performed according to each sub-segmentation interval.

[0010] According to a preferred embodiment of the present invention, the data type of the primary key of the data table includes: int type and / or bigint type; the virtual split value is the maximum value of int unsigned in the data table.

[0011] According to a preferred embodiment of the present invention, slicing the primary key of each partition interval to obtain multiple sub-slice intervals includes:

[0012] Determine the split value based on the maximum primary key value, minimum primary key value and number of slices within the split interval;

[0013] The split interval is divided into multiple sub-split intervals according to the split value.

[0014] According to a preferred embodiment of the present invention, synchronizing data according to each sub-segment interval includes:

[0015] Assign the data corresponding to the primary key in each sub-segment interval to the corresponding data synchronization task;

[0016] Synchronously execute various data synchronization tasks.

[0017] According to a preferred embodiment of the present invention, before configuring the virtual split value according to the data type of the data table primary key, the method further includes:

[0018] Create the original data table;

[0019] Receive a data table partitioning instruction, and change the original data table primary key to the primary key of the data table after partitioning; the primary key of the data table after partitioning includes: int type primary key and / or bigint type primary key.

[0020] According to a preferred embodiment of the present invention, a new primary key is added to the original data table int type primary key to obtain the primary key of the data table after partitioning; wherein: the new primary key is generated using a snowflake algorithm.

[0021] To solve the above technical problems, the second aspect of the present invention provides a data synchronization device based on slicing processing, the device comprising:

[0022] Configuration module, used to configure virtual split values ​​according to the data type of the primary key of the data table;

[0023] A segmentation module, configured to divide the data table primary key into a plurality of segmentation intervals based on the virtual segmentation value;

[0024] The slicing module is used to slice the primary key of each partition interval to obtain multiple sub-slice intervals;

[0025] The synchronization module is used to synchronize data according to each sub-segment interval.

[0026] According to a preferred embodiment of the present invention, the data type of the data table primary key includes: int type and / or bigint type; the configuration module configures the virtual split value to be the maximum value of int unsigned in the data table.

[0027] According to a preferred embodiment of the present invention, the slicing module includes:

[0028] A determination module is used to determine the split value according to the maximum primary key value, the minimum primary key value and the number of slices in the split interval;

[0029] The sub-segmentation module is used to divide the segmentation interval into multiple sub-segmentation intervals according to the segmentation value.

[0030] According to a preferred embodiment of the present invention, the synchronization module includes:

[0031] The sub-configuration module is used to configure the data corresponding to the primary key in each sub-segment interval to the corresponding data synchronization task;

[0032] The sub-synchronization module is used to synchronously execute various data synchronization tasks.

[0033] According to a preferred embodiment of the present invention, the device further comprises:

[0034] Create a module to create the original data table;

[0035] The database sharding module is used to receive a data table sharding instruction and change the original data table primary key to the primary key of the data table after sharding; the primary key of the data table after sharding includes: int type primary key and / or bigint type primary key.

[0036] According to a preferred embodiment of the present invention, the database sharding module adds a new primary key based on the int-type primary key of the original data table to obtain the primary key of the data table after sharding; wherein: the new primary key is generated using a snowflake algorithm.

[0037] To solve the above technical problems, the present invention provides an electronic device according to a third aspect, including:

[0038] processor; and

[0039] A memory storing computer executable instructions, which, when executed, cause the processor to perform the above method.

[0040] In order to solve the above technical problems, the fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the above method is implemented.

[0041] The present invention configures a virtual split value according to the data type of the primary key of the data table, and divides the primary key into multiple split intervals according to the virtual split value; since the virtual split value can distinguish the primary keys of different continuous storage ranges, the primary keys of the same continuous storage range can be divided into the same split interval through the virtual split value, so that each split interval only contains the primary keys of the same continuous storage range. In this way, the primary key values ​​of each split interval are continuous and there is no jump. Then the primary key of each split interval is sliced ​​to ensure that the number of data items contained in each sub-slice interval in each split interval is the same after the slicing process, thereby avoiding data skew, thereby reducing data synchronization time and improving data synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to make the technical problems solved by the present invention, the technical means adopted, and the technical effects achieved more clearly, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention. Those skilled in the art can derive drawings of other embodiments based on these drawings without inventive effort.

[0043] Figure 1 This is a flow chart of a data synchronization method based on slicing processing according to an embodiment of the present invention;

[0044] Figure 2 It is adopted Figure 1 Schematic diagram of data synchronization based on slice processing for data table A containing int and bigint primary keys;

[0045] Figure 3 This is a schematic diagram showing how to directly slice table A, which only contains an int primary key, to obtain multiple sub-partitions.

[0046] Figure 4 This is a schematic diagram of directly slicing data table B containing int and bigint primary keys to obtain multiple sub-partitions;

[0047] Figure 5 This is a comparison diagram of the time required for synchronization before and after test data optimization;

[0048] Figure 6 This is a schematic diagram of the structural framework of a data synchronization device based on slicing processing according to an embodiment of the present invention;

[0049] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention;

[0050] Figure 8 is a schematic diagram of an embodiment of a computer-readable medium of the present invention. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the content of the present invention more complete and to more fully convey the inventive concept to those skilled in the art.

[0052] Under the premise of being consistent with the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0053] In the introduction of specific embodiments, the detailed description of the structure, performance, effect or other features is intended to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from implementing the present invention with a technical solution that does not include the aforementioned structure, performance, effect or other features under specific circumstances.

[0054] The flowcharts in the accompanying drawings are merely illustrative of the process flow and do not necessarily include all of the content, operations, and steps in the flowcharts, nor do they necessarily imply that all of the steps in the flowcharts must be executed in the order shown. For example, some of the steps in the flowcharts may be separated, some may be combined or partially combined, and so on. The execution order shown in the flowcharts may be changed according to actual circumstances without departing from the spirit of the present invention.

[0055] Frames in the accompanying drawings Figure 1 The term "functional entity" generally refers to a functional entity and does not necessarily correspond to a physically independent entity. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The same reference numerals in the accompanying drawings represent the same or similar elements, components or parts, and thus repeated descriptions of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other numbered adjectives may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these adjectives. In other words, these adjectives are only used to distinguish one from another. For example, the first device may also be called the second device, but this does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0057] See also Figure 1 , Figure 1 This invention provides a data synchronization method based on slicing processing, such as Figure 1 As shown, the method includes:

[0058] S1. Configure the virtual split value according to the data type of the primary key of the data table;

[0059] In an embodiment of the present invention, data types are distinguished based on the continuous storage range of integer data. Therefore, different data types correspond to different continuous storage ranges of integer data. Within the same continuous storage range, the size of the integer data increases sequentially without jumps. For example, data types may include int, bigint, smallint, and so on. Among them, int corresponds to storing all integer data between -2147483648 and 2147483647, with a storage size of 4 bytes; bigint corresponds to storing all integer data between -9223372036854775808 and 9223372036854775807, with a storage size of 8 bytes; smallint corresponds to storing all integer data between -32768 and 32767, with a storage size of 2 bytes.

[0060] In this embodiment, if the primary key of a data table contains at least two different data types, the primary keys belong to at least two different continuous storage ranges, and the primary key values ​​may jump. The present invention configures a virtual split value for the data type of the primary key of the data table, wherein: the virtual split value is used to distinguish primary keys of different continuous storage ranges; the virtual split value can be used to divide primary keys of the same continuous storage range into the same split interval, so that each split interval only contains primary keys of the same continuous storage range. The primary keys of each split interval are then sliced ​​to ensure that the number of data entries contained in each sub-slice interval within each split interval after slicing is the same, thus avoiding data skew.

[0061] Exemplarily, the virtual split value can be configured as a demarcation value for different data types contained in the primary key. The demarcation value can be the maximum value corresponding to a data type with fewer storage bits, or the minimum value corresponding to a data type with more storage bits. For example: the data types of the primary key include: int type and / or bigint type, int type is 32 bits, and bigint type is 64 bits, then the virtual split value can be configured as the maximum value corresponding to int type, or the minimum value corresponding to bingint type. For another example: in a relational database, the data types of the primary key include: int type and bigint type, then the virtual split value can be configured as the maximum value of unsigned int in the relational database, that is: 4294967295.

[0062] S2. Divide the data table primary key into multiple segmentation intervals based on the virtual segmentation value;

[0063] In this embodiment of the present invention, a virtual split value is used to distinguish primary keys from different continuous storage ranges. Primary keys within the same continuous storage range are grouped into the same split interval using the virtual split value, so that each split interval contains only primary keys from the same continuous storage range. Optionally, the virtual split value serves as a delimiter between different data types within the primary key. This delimiter can be used to divide the primary key into multiple split intervals.

[0064] like Figure 2 Table A's primary key data types include int and bigint. Int primary key IDs range from 1 to 5000, while bigint primary key IDs are 6068631590756503554. The virtual split value is the maximum value of intunsigned in a relational database, which is 4294967295. The virtual split value 4294967295 divides the primary key range 1 to 5000 into split range A, corresponding to the int primary key, and divides 6068631590756503554 into split range B, corresponding to the bigint primary key.

[0065] S3. Slice the primary key of each partition interval to obtain multiple sub-slice intervals.

[0066] In this embodiment, the slicing process for the primary key of each partition interval may be performed synchronously or asynchronously, wherein the slicing process for the primary key of each partition interval to obtain multiple sub-slicing intervals may include:

[0067] S31, determining a split value based on the maximum primary key value, the minimum primary key value, and the number of slices within the split interval;

[0068] When the maximum value of the primary key between the slices is not equal to the minimum value of the primary key, the slice value is (maximum value of the primary key - minimum value of the primary key) / number of slices, and then rounded up. Figure 2 In the example, the primary key is divided into partitions A and B using the virtual partition value 4294967295. For partition A, the primary key minimum value is (1), the primary key maximum value is (5000), and the partition digit is 2. Then, (5000-1) / 2 is 2499.5, which is then rounded up to obtain a partition value of 2500.

[0069] When the maximum value of the primary key between partitions is equal to the minimum value of the primary key, there is no split value in the split interval. For example Figure 2 In the example, for split interval B, the primary key minimum value is equal to the primary key maximum value, and both are 6068631590756503554. Therefore, split interval B does not exist in the split value.

[0070] S32: Divide the segmentation interval into multiple sub-segmentation intervals according to the segmentation value.

[0071] If there is a split value, the split value can be used as a dividing line to divide the primary key of the split interval into the corresponding sub-split intervals. Figure 2 , the split value 2500 divides the split interval A (primary key ID: 1-5000) into sub-split interval a (primary key ID: 1-2500) and sub-split interval b (primary key ID: 2500-5000). If there is no split value, the split interval is not divided and the split interval is directly used as a sub-split interval. For example Figure 2 , segmentation interval B does not exist in the segmentation value, and the primary key ID of segmentation interval B: 6068631590756503554 is directly used as the sub-segmentation interval c.

[0072] S4. Synchronize data according to each sub-segmentation interval.

[0073] For example, in this step, the data corresponding to the primary key in each sub-segment interval can be configured into the corresponding data synchronization task, and then each data synchronization task can be executed synchronously. Figure 2 , configure the data corresponding to the primary key ID: 1-2500 in sub-segment interval a to the data synchronization task task1, configure the data corresponding to the primary key ID: 2500-5000 in sub-segment interval b to the data synchronization task task2, configure the data corresponding to the primary key ID: 6068631590756503554 in sub-segment interval c to the data synchronization task task3, and control the synchronous execution of task1 to task3. Among them, task1 synchronized 2500 pieces of data, task2 synchronized 2500 pieces of data, and task3 synchronized 1 piece of data due to the limitation of data volume. It can be seen that the present invention can ensure that in the main data segment, data is evenly synchronized between each task, and data skew will not occur, thereby reducing data synchronization time and improving data synchronization efficiency.

[0074] In this embodiment, the data type of the data table primary key may include: int, bigint, or int and bigint. The virtual split value can be configured as the maximum value of int unsigned in the data table to avoid data skew.

[0075] In actual business, the primary key of the corresponding data type can be configured according to the size of the business data. For example: if the business data volume is less than the storage capacity of 4 bytes, the primary key data type can be configured as a 4-byte int type to meet the data storage requirements. In this case, the primary key belongs to the same continuous storage range, the primary key size increases in sequence, and there is no primary key value jump. Then, the data table primary key can be directly sliced ​​to obtain multiple sub-slicing intervals, and then step S4 is executed to achieve data synchronization while ensuring that the number of data items contained in each sub-slicing interval after slicing is the same, avoiding data skew.

[0076] In the process of directly slicing the primary key of the data table to obtain multiple sub-slicing intervals, for example Figure 3 In the example, data table A contains 10,000 records. The primary key data type is int, ranging from 1 to 10,000. Taking the number of slices as 2 as an example, during the slicing process, the minimum value (1) and the maximum value (10,000) of the primary key of data table A are first obtained. Then, the maximum value (10) of the primary key is divided by the minimum value (1) of the primary key / the number of slices (2) to get 4999.5, which is rounded up to get the slice value of 5000. Based on the slice value, sub-slicing interval 1 (primary key ID from 1 to 5000) and sub-slicing interval 2 (primary key ID from 5000 to 10000) are obtained. Since the data type of the primary key is the same, the primary key ID increases with the number of data records, and there is no jump in the primary key ID. This ensures that the number of data records in the sub-slicing intervals obtained by slicing is the same. In this example, sub-slicing interval 1 and sub-slicing interval 2 contain the same number of data records, both 5000. During the data synchronization process, 5,000 data items in slice interval 1 can be assigned to data synchronization task task1, and 5,000 data items in slice interval 2 can be assigned to data synchronization task task2. In this way, each data synchronization task synchronizes 5,000 data items, avoiding data skew.

[0077] In the embodiment of the present invention, an int-type primary key can be configured when creating a data table, and the int-type primary key and the bigint-type primary key can be obtained by adding or modifying the int-type primary key according to the storage requirements of the business data volume.

[0078] Therefore, in one example, before step S1, the method may further include:

[0079] S11. Configure an int primary key and create the original data table;

[0080] For example, in the early stages of a business, when the amount of data is small, you can use an int primary key to create the original data table. However, as the business develops, the amount of data gradually increases, and a single database and a single table are not sufficient to meet the storage needs of business data. The original data table can be split into separate databases.

[0081] S12. Receive a data table partitioning instruction, and modify the original data table primary key to the primary key of the data table after partitioning. The primary key of the data table after partitioning includes: an int type primary key and / or a bigint type primary key.

[0082] Exemplarily, during the database sharding process, sharding rules can be preconfigured. Upon receiving a data table sharding instruction, the original database and data table are split according to the sharding rules. Data originally stored in one database is split and stored across multiple databases, and data originally stored in one table is split and stored across multiple tables. While splitting the data, the original data table primary key is modified to the primary key of the data table after sharding. The primary key of the data table after sharding includes an int type primary key and / or a bigint type primary key.

[0083] In one addition and modification method, a new primary key can be added based on the int-type primary key of the original data table to obtain the primary key of the data table after the database is divided. The new primary key can be generated using the SnowFlake algorithm. The SnowFlake algorithm is a distributed ID generation algorithm that can generate a unique ID that increases according to the time trend. The ID mainly consists of four parts: the first invalid character, the timestamp difference, the machine code, and the serial number. It is a 64-bit bigint-type ID. Therefore, using the ID generated by the SnowFlake algorithm as the new primary key can ensure that the new primary key is self-increasing and unique.

[0084] During the database sharding process, the primary key type is changed from an int to a bigint type generated by the snowflake algorithm, which will cause data skew during data synchronization and make data synchronization extremely slow. For example: Figure 4 In the example, the original data table is changed from the first primary key of int type to the data tables B and C after the partition, which include the first primary key and the new primary key generated by the snowflake algorithm. The primary key ID of data table B includes: int type (primary key ID value ranges from 1 to 5000) and bigint type (primary key ID value ranges from 6068631590756503554). Figure 3 To directly slice the primary key of table B to obtain multiple sub-slice intervals, first calculate the minimum value (1) and maximum value (6068631590756503554) of the primary key, and then use the maximum value (6068631590756503554) - minimum value (1) / number of slices (for example, the number of slices is 2) to obtain the slice value 3034315795378251777, so that sub-slice interval 1 (primary key 1-3034315795378251777) corresponds to data synchronization task task1; sub-slice interval 2 (primary key 3034315795378251777-6068631590756503554) corresponds to data synchronization task task2.

[0085] like Figure 4 For data table B, sub-shard interval 1 corresponding to task 1 can process 5,000 data items, while sub-shard interval 2 corresponding to task 2 can only process 1 data item, resulting in data skew. As a result, during the data synchronization process, task 2 may only synchronize in a few minutes, while task 1 may take 30 minutes or even an hour. This leads to slow data synchronization, late data readiness in the data warehouse, and delayed downstream tasks.

[0086] It can be seen that after step S12, since the primary key of the data table after sharding contains multiple continuous storage ranges, directly slicing the primary key will cause data skew during data synchronization. To solve the problem of data skew, steps S1 to S4 need to be executed after step S12.

[0087] More than 20 million data records are stored in a data table containing an int primary key and a new primary key generated by the snowflake algorithm as test data. Figure 5 This is a comparative diagram of the time required for synchronization before test data optimization (i.e., directly slicing to achieve data synchronization) and the time required for synchronization after optimization (i.e., using the data synchronization method based on slicing processing of the present invention to achieve data synchronization). Figure 5 It can be seen that the data synchronization time of a data table with multiple data types and primary keys containing more than 20 million data records was 1680 seconds before optimization. After optimization, the data skew caused by multiple data types and primary keys was resolved, and the data synchronization time was reduced to 457 seconds, increasing data synchronization efficiency by 3.6 times.

[0088] Figure 6 The present invention is a data synchronization device based on slicing processing, such as Figure 6 As shown, the device includes:

[0089] Configuration module 61, configured to configure a virtual split value according to the data type of the primary key of the data table;

[0090] A segmentation module 62 is configured to divide the data table primary key into a plurality of segmentation intervals based on the virtual segmentation value;

[0091] Slicing module 63, used for slicing the primary key of each partition interval to obtain multiple sub-slice intervals;

[0092] The synchronization module 64 is used to synchronize data according to each sub-segment interval.

[0093] In one embodiment, the data type of the primary key of the data table includes: int type and / or bigint type; the configuration module 61 configures the virtual split value to be the maximum value of int unsigned in the data table.

[0094] The slicing module 63 includes:

[0095] A determination module is used to determine the split value according to the maximum primary key value, the minimum primary key value and the number of slices in the split interval;

[0096] The sub-segmentation module is used to divide the segmentation interval into multiple sub-segmentation intervals according to the segmentation value.

[0097] The synchronization module 64 includes:

[0098] The sub-configuration module is used to configure the data corresponding to the primary key in each sub-segment interval to the corresponding data synchronization task;

[0099] The sub-synchronization module is used to synchronously execute various data synchronization tasks.

[0100] Furthermore, the device further comprises:

[0101] Create a module to create the original data table;

[0102] The database sharding module is used to receive a data table sharding instruction and change the original data table primary key to the primary key of the data table after sharding; the primary key of the data table after sharding includes: int type primary key and / or bigint type primary key.

[0103] Wherein: the database sharding module adds a new primary key based on the int-type primary key of the original data table to obtain the primary key of the data table after sharding; wherein: the new primary key is generated using the snowflake algorithm.

[0104] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be modified accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module or further split into multiple submodules.

[0105] The following describes an electronic device embodiment of the present invention. This electronic device can be considered a physical implementation of the method and apparatus embodiments of the present invention described above. Details described in the electronic device embodiment of the present invention should be considered supplementary to the above-described method or apparatus embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above-described method or apparatus embodiments.

[0106] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0107] like Figure 7As shown, the electronic device 700 of this exemplary embodiment is in the form of a general-purpose data processing device. Components of the electronic device 700 may include, but are not limited to, at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different electronic device components (including the storage unit 720 and the processing unit 710), a display unit 740, and the like.

[0108] The storage unit 720 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 710, so that the processing unit 710 performs the steps of various embodiments of the present invention. For example, the processing unit 710 may perform the following steps: Figure 1 Steps shown.

[0109] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202, and may further include a read-only memory unit (ROM) 7203. The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: operating the electronic device, one or more application programs, other program modules, and program data, each of which or some combination thereof may include the implementation of a network environment.

[0110] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0111] The electronic device 700 can also communicate with one or more external devices 100 (e.g., keyboard, display, network device, Bluetooth device, etc.), so that the user can interact with the electronic device 700 via these external devices 100, and / or enable the electronic device 700 to communicate with one or more other data processing devices (e.g., router, modem, etc.). Such communication can be carried out through the input / output (I / O) interface 750, and can also be carried out with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through the network adapter 760. The network adapter 760 can communicate with other modules of the electronic device 700 through the bus 730. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronic devices, tape drives, and data backup storage electronic devices.

[0112] Figure 8 Schematic diagram of a computer readable medium embodiment of the present invention. Figure 8 As shown, the computer program can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic device, apparatus, or device that is electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. When the computer program is executed by one or more data processing devices, the computer-readable medium is enabled to implement the above-mentioned method of the present invention, namely: configuring a virtual split value based on the data type of the data table primary key; dividing the data table primary key into multiple split intervals based on the virtual split value; slicing the primary key of each split interval to obtain multiple sub-split intervals; and synchronizing data according to each sub-split interval.

[0113] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes several instructions to enable a data processing device (which can be a personal computer, server, or network device, etc.) to execute the above method according to the present invention.

[0114] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0115] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0116] In summary, the present invention can be implemented by a method, apparatus, electronic device or computer-readable medium that executes a computer program. In practice, a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all of the functions of the present invention.

[0117] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A data synchronization method based on slicing processing, characterized in that: The method comprises: A virtual split value is configured according to the data type of the data table primary key; the data table primary key includes at least two different data types, the data types are distinguished according to the continuous storage range of integer data, different data types correspond to different continuous storage ranges of integer data, and within the same continuous storage range, the size of the integer data increases in sequence without jumps; the virtual split value is used to distinguish primary keys of different continuous storage ranges; primary keys of the same continuous storage range can be divided into the same split interval by the virtual split value, so that each split interval only contains primary keys of the same continuous storage range; Dividing the data table primary key into multiple segmentation intervals based on the virtual segmentation value; Slice the primary key of each partition interval to obtain multiple sub-partition intervals; Data synchronization is performed according to each sub-segmentation interval.

2. The method according to claim 1, characterized in that The data type of the primary key of the data table includes: int type and / or bigint type; the virtual split value is the maximum value of int unsigned in the data table.

3. The method according to claim 1, characterized in that The primary key of each partition is sliced ​​to obtain multiple sub-slice intervals including: Determine the split value based on the maximum primary key value, minimum primary key value and number of slices within the split interval; The split interval is divided into multiple sub-split intervals according to the split value.

4. The method according to claim 1, wherein The data synchronization according to each sub-segment interval includes: Assign the data corresponding to the primary key in each sub-segment interval to the corresponding data synchronization task; Synchronously execute various data synchronization tasks.

5. The method according to any one of claims 1 to 4, characterized in that Before configuring the virtual split value according to the data type of the data table primary key, the method further includes: Create the original data table; Receive a data table partitioning instruction, and change the original data table primary key to the primary key of the data table after partitioning; the primary key of the data table after partitioning includes: int type primary key and / or bigint type primary key.

6. The method according to claim 5, characterized in that Based on the int-type primary key of the original data table, a new primary key is added to obtain the primary key of the data table after the database is divided; wherein: the new primary key is generated using the snowflake algorithm.

7. A data synchronization device based on slicing processing, characterized in that: The device comprises: A configuration module is configured to configure a virtual split value according to a data type of a data table primary key; the data table primary key includes at least two different data types, the data types are distinguished according to a continuous storage range of integer data, different data types correspond to different continuous storage ranges of integer data, and within the same continuous storage range, the size of the integer data increases sequentially without jumps; the virtual split value is used to distinguish primary keys of different continuous storage ranges; primary keys of the same continuous storage range can be divided into the same split interval by using the virtual split value, so that each split interval only contains primary keys of the same continuous storage range; A segmentation module, configured to divide the data table primary key into a plurality of segmentation intervals based on the virtual segmentation value; The slicing module is used to slice the primary key of each partition interval to obtain multiple sub-slice intervals; The synchronization module is used to synchronize data according to each sub-segment interval.

8. The device according to claim 7, characterized in that The data type of the primary key of the data table includes: int type and / or bigint type; the configuration module configures the virtual split value to be the maximum value of int unsigned in the data table.

9. The device according to claim 7, characterized in that The slicing module includes: A determination module is used to determine the split value according to the maximum primary key value, the minimum primary key value and the number of slices in the split interval; The sub-segmentation module is used to divide the segmentation interval into multiple sub-segmentation intervals according to the segmentation value.

10. The device according to claim 7, characterized in that The synchronization module includes: The sub-configuration module is used to configure the data corresponding to the primary key in each sub-segment interval to the corresponding data synchronization task; The sub-synchronization module is used to synchronously execute various data synchronization tasks.

11. The device according to any one of claims 7 to 10, characterized in that: The device further comprises: Create a module to create the original data table; The database sharding module is used to receive a data table sharding instruction and change the original data table primary key to the primary key of the data table after sharding; the primary key of the data table after sharding includes: int type primary key and / or bigint type primary key.

12. The device according to claim 11, characterized in that The database sharding module adds a new primary key based on the int-type primary key of the original data table to obtain the primary key of the data table after the database sharding; wherein: the new primary key is generated using a snowflake algorithm.

13. An electronic device comprising: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images