A data table synchronization method, synchronization device and storage medium based on Spark framework

Abstract

The present application discloses a data table synchronization method, synchronization device and storage medium based on the Spark framework. Based on the present application, the number of metadata in the data table to be processed is obtained based on the Spark framework, and an auto-increment field is generated for each metadata to form an auto-increment field table; and based on the data processing capacity of each data partition and the average length of the metadata to be processed, the average processable data volume of the data partition is calculated. Further, the auto-increment field table is divided based on the average processable data volume to complete the division of the data partitions corresponding to each metadata in the data table to be processed, so that the data in each data partition is uniform and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data table synchronization method, a synchronization device, and a storage medium based on a Spark framework. Background Art

[0002] In the process of synchronizing business data tables of a relational database management system (hereinafter referred to as MySQL database) with a certain scale, if the table contains uneven primary key distribution or non-integer primary key, data skew and synchronization failure may occur. Among them, in the process of Apache Spark computing engine (hereinafter referred to as Spark framework) concurrently pulling business data tables of MySQL database, it is necessary to partition according to an integer field (usually the primary key). The existing solution is to add a column of auto-increment integer field when pulling data to meet the pull requirements of Spark framework, but this method will generate a large number of temporary files in the nodes of MySQL database. At the same time, partitioning by non-integer field is not supported, and a column of auto-increment integer field needs to be added to meet the pull requirements of Spark framework, but subqueries will be generated. Subqueries will generate a large number of temporary files in the nodes of MySQL database. The larger the data volume and the higher the concurrency, the larger the temporary files generated, which will cause MySQL database abnormal unavailability.

[0003] In addition, when processing data, the upper and lower limits of the auto-increment integer field passed in are split into several intervals of the same size by default according to the number of partitions. Since there are many deletion operations involved in the MySQL database, some intervals do not get data, some nodes are "running in vain", and some nodes are "busy". Data skew not only wastes resources and takes a long time to run tasks, but may also cause task failures. Summary of the invention

[0004] The embodiments of the present application provide a data table synchronization method, a synchronization device, and a storage medium based on a Spark framework, which help to improve data processing efficiency.

[0005] In one embodiment, a data table synchronization method based on the Spark framework includes:

[0006] Acquire at least one primary key of metadata in the data table to be processed based on the Spark framework, and generate an auto-increment field table of auto-increment fields corresponding to at least one primary key in the data table to be processed based on the number of acquired primary keys;

[0007] Based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed, the average processable data volume of the data partition is calculated;

[0008] The auto-increment field table is divided according to the auto-increment field table and the average processable data volume, and the data partition where at least one primary key is located is divided based on the auto-increment field contained in at least one sub-auto-increment field table after the division, and each metadata corresponding to the primary key in the data table to be processed after the division is synchronized to the corresponding data partition.

[0009] Optionally, register the obtained primary key as a primary key temporary table;

[0010] Starting with the initial primary key in the primary key temporary table, and based on the number of primary keys obtained, a corresponding auto-increment field is generated for each primary key in the temporary table, and an auto-increment field table corresponding to the primary key temporary table is generated.

[0011] Optionally, traverse at least one auto-increment field in the auto-increment field table, perform a modulo operation on at least one auto-increment field and the average processable data amount, and when there is a remainder and the remainder is 0, use the corresponding auto-increment field as a division point, and divide the auto-increment field table into sub-auto-increment field tables of the same number as the number of division points.

[0012] Optionally, a corresponding data partition is matched for each sub-auto-increment field table, and a primary key corresponding to the auto-increment field in each sub-auto-increment field is divided into a corresponding data partition;

[0013] Based on the primary key corresponding to each data partition, the metadata in the to-be-processed data table corresponding to the primary key is concurrently pulled, and the metadata is synchronized to the corresponding data partition.

[0014] In another embodiment, a data table synchronization device based on a Spark framework is provided, the synchronization device comprising:

[0015] An acquisition module is used to acquire a primary key of at least one metadata in a to-be-processed data table based on a Spark framework, and generate an auto-increment field table of auto-increment fields corresponding to at least one primary key in the to-be-processed data table based on the acquired number of primary keys;

[0016] A calculation module, used to calculate an average processable data volume of each data partition based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed;

[0017] A synchronization module is used to divide the auto-increment field table according to the auto-increment field table and the average processable data volume, and divide the data partition where at least one primary key is located based on the auto-increment field contained in at least one sub-auto-increment field table after the division, and synchronize the metadata corresponding to the primary key in the data table to be processed after the division to the corresponding data partition.

[0018] Optionally, the acquisition module is also used to:

[0019] Register the obtained primary key into the primary key temporary table;

[0020] Starting with the initial primary key in the primary key temporary table, and based on the number of primary keys obtained, a corresponding auto-increment field is generated for each primary key in the temporary table, and an auto-increment field table corresponding to the primary key temporary table is generated.

[0021] Optionally, the synchronization module is further used to:

[0022] Traverse at least one auto-increment field in the auto-increment field table, perform a modulo operation on at least one auto-increment field and the average processable data volume, and when there is a remainder and the remainder is 0, use the corresponding auto-increment field as a division point, and divide the auto-increment field table into sub-auto-increment field tables with the same number as the number of division points.

[0023] Optionally, the synchronization module is further used to:

[0024] Match the corresponding data partition for each sub-auto-increment field table, and divide the primary key corresponding to the auto-increment field in each sub-auto-increment field into the corresponding data partition;

[0025] Based on the primary key corresponding to each data partition, the metadata in the to-be-processed data table corresponding to the primary key is concurrently pulled, and the metadata is synchronized to the corresponding data partition.

[0026] In another embodiment of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to execute the data table synchronization method based on the Spark framework in the aforementioned embodiment.

[0027] In another embodiment, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the data table synchronization method based on the Spark framework described in any one of the aforementioned embodiments.

[0028] In another embodiment, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the data table synchronization method based on the Spark framework as described in any one of the aforementioned embodiments.

[0029] Based on the above embodiment, the number of metadata in the data table to be processed is obtained based on the Spark framework, and an auto-increment field is generated for each metadata to form an auto-increment field table; and based on the data processing capacity of each data partition and the average length of the metadata to be processed, the average processable data volume of the data partition is calculated. Further, the auto-increment field table is divided based on the average processable data volume to complete the division of the data partitions corresponding to each metadata in the data table to be processed, so that the data in each data partition is uniform and the data processing efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of the principle of a data table synchronization method based on the Spark framework according to an embodiment of the present application;

[0032] Figure 2 For Figure 1 A schematic diagram of the process of the data table synchronization method based on the Spark framework in the illustrated embodiment;

[0033] Figure 3 This is a schematic diagram of an extended process of a data table synchronization method based on a Spark framework in another embodiment of the present application;

[0034] Figure 4 The figure shows a data flow diagram of a data table synchronization method based on the Spark framework in one embodiment of the present application;

[0035] Figure 5 This is a schematic diagram of a data table synchronization device based on the Spark framework in another embodiment of the present application;

[0036] Figure 6 A schematic diagram of an electronic device provided according to another embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the implementation of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] The embodiment of the present application can be used to divide the data table with non-integer primary key and uneven primary key distribution in the MySQL database into custom data partitions for the Spark framework synchronization under the background of large task volume, so as to realize the uniform distribution of metadata in the data partitions. Figure 1 As shown, it is a schematic diagram of the principle of a data table synchronization method based on the Spark framework shown in an embodiment of the present application. Among them, based on the Spark framework, the primary key of at least one metadata contained in the data table to be processed in the MySQL database is obtained, and each primary key corresponds to the metadata one by one as the keyword data of the corresponding metadata. After pulling and arranging the obtained primary keys (primary key 1, primary key 3...primary key N), a primary key temporary table is generated. For each primary key, an auto-increment field is generated, and each primary key corresponds to an auto-increment field (primary key 1 corresponds to auto-increment field 1, primary key 3 corresponds to auto-increment field 3...primary key N corresponds to auto-increment field 3), forming an auto-increment field table. Further, the data processing capacity of each data partition in the Hadoop cluster distributed system (hereinafter referred to as the Hadoop system) is obtained, and the average length of each metadata in the data table to be processed is calculated, and the average processable data volume of the data partition is calculated based on the obtained data processing capacity and average length. The auto-increment field table is divided based on the average processable data volume, so that each data partition corresponds to a sub-auto-increment field table. And, the metadata is divided accordingly based on the primary key corresponding to the corresponding auto-increment field in each sub-auto-increment field table, so as to synchronize the metadata to the corresponding data partition.

[0040] Figure 2 To achieve Figure 1 The flowchart of the data table synchronization method based on the Spark framework is shown in FIG. Figure 2 As shown, the specific process of the synchronization method may include the following steps:

[0041] S101, obtaining a primary key of at least one metadata in a to-be-processed data table based on a Spark framework, and generating an auto-increment field table of auto-increment fields corresponding to at least one primary key in the to-be-processed data table based on the number of obtained primary keys.

[0042] In this step, the data table to be processed obtained by the embodiment of the present application is mainly a data table to be processed stored in a MySQL database. Based on the Spark framework, at least one metadata in the data table to be processed can be concurrently obtained. Among them, the keyword data of each metadata is obtained, that is, the primary key. After the primary keys of the metadata are concurrently obtained based on the Spark framework, according to the number of primary keys obtained, the same number of auto-increment fields corresponding to each primary key are generated to form an auto-increment field table. The values ​​corresponding to the auto-increment fields in the auto-increment field table are continuous and increase bit by bit, and are integer fields.

[0043] S102, calculating the average processable data volume of each data partition based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed.

[0044] In this step, based on the data processing capability of the data partition contained in the Hadoop system, i.e., the data capacity of the data partition, and the average length of the metadata in the data table to be processed, the amount of metadata that can be processed by the data partition is calculated as the average processable data volume of the data partition.

[0045] S103, dividing the auto-increment field table according to the auto-increment field table and the average processable data volume, and dividing the data partition where at least one primary key is located based on the auto-increment field contained in at least one sub-auto-increment field table after the division, and synchronizing each metadata corresponding to the primary key in the data table to be processed after the division to the corresponding data partition.

[0046] In this step, the auto-increment field table is divided based on the average processable data volume of the data partition obtained above, that is, the auto-increment field table is divided into at least one sub-auto-increment field table, and the number of auto-increment fields contained in each sub-auto-increment field table is the same as the average processable data volume. And, based on the primary key corresponding to the auto-increment field contained in the sub-auto-increment field table, each metadata in the to-be-processed data table corresponding to the primary key is pulled, and each metadata is synchronized to the corresponding data partition.

[0047] Based on the collection method of the above embodiment, the number of metadata in the data table to be processed is obtained based on the Spark framework, and an auto-increment field is generated for each metadata to form an auto-increment field table; and based on the data processing capacity of each data partition and the average length of the metadata to be processed, the average processable data volume of the data partition is calculated. Further, the auto-increment field table is divided based on the average processable data volume to complete the division of the data partitions corresponding to each metadata in the data table to be processed, so that the data in each data partition is uniform and the data processing efficiency is improved.

[0048] Figure 3 For Figure 2 The extended process diagram of the data table synchronization method based on the Spark framework is shown in the figure. Figure 3 , the data table synchronization method based on the Spark framework can be extended to include the following steps:

[0049] S301, obtaining at least one primary key of metadata in a data table to be processed based on the Spark framework.

[0050] In this step, the Spark framework is a fast and versatile computing engine designed for large-scale data processing. The embodiment of the present application uses the Spark framework to separately pull the primary key of the metadata in the data table to be processed in the MySQL database. When pulling the primary key, the Spark framework does not generate a temporary file and only pulls the primary key data, not the metadata. Therefore, the concurrency of the Spark framework when pulling the primary key can be adjusted based on the resource scheduling of the Hadoop system that needs to process the data table to be processed, thereby improving the pulling efficiency.

[0051] S302, registering the obtained primary key as a primary key temporary table.

[0052] In this step, the obtained primary keys are arranged evenly and registered as a temporary primary key table based on the Hadoop system's data warehouse tool hive.

[0053] S303, starting with the initial primary key in the primary key temporary table, and based on the acquired number of primary keys, generating corresponding auto-increment fields for each primary key in the temporary table, generating an auto-increment field table corresponding to the primary key temporary table.

[0054] In this step, each auto-increment field in the auto-increment field table corresponds to a primary key. Starting from the initial primary key corresponding to the initial auto-increment field, the window function of the primary key temporary table registered by the data warehouse tool hive is used to generate an auto-increment field for each primary key in the primary key temporary table. The value corresponding to the auto-increment field is continuous and increases bit by bit, and is an integer field. The generation process of the auto-increment field table is performed in the data warehouse tool hive and will not affect the MySQL database.

[0055] S304, obtaining the data processing capability of each data partition and the average length of each metadata in the data table to be processed.

[0056] In this step, the block size blockSize of each data partition in the Hadoop system is obtained, that is, the number of data that the data partition can process. Among them, the actual data processing capacity of each data partition can be defined based on the block size blockSize of each data partition. Considering the computing power of the manager node nameNode in the Hadoop system, if the block size of the data partition is known to be blockSize, the data processing capacity that each data partition can actually process can be defined as maxSize=XblockSize. X is a decimal between 0 and 1. If the size of a data partition in the Hadoop system is 256M, it can be defined that the data partition can process 200M of data.

[0057] And, the source data information of the data table to be processed in the MySQL database is obtained, which includes the data length of each metadata, and the average length avgSize of each metadata in the data table to be processed is calculated based on the data length.

[0058] S305, calculating the average processable data volume of the data partition.

[0059] In this step, based on the data processing capacity maxSize of the data partition obtained above and the average length avgSize of the metadata, the average processable data volume p of the data partition is calculated = maxSize / avgSize.

[0060] S306, dividing the auto-increment field table according to the auto-increment field table and the average processable data volume.

[0061] In this step, at least one auto-increment field in the auto-increment field table is traversed, and a modulo operation is performed on at least one auto-increment field and the average processable data volume. When there is a remainder and the remainder is 0, the corresponding auto-increment field is used as a division point, and the auto-increment field table is divided into sub-auto-increment field tables with the same number as the number of division points.

[0062] S307, matching a corresponding data partition for each sub-auto-increment field table, and dividing the primary key corresponding to the auto-increment field in each sub-auto-increment field into the corresponding data partition.

[0063] S308, based on the primary key corresponding to each data partition, concurrently pull the metadata in the to-be-processed data table corresponding to the primary key, and synchronize the metadata to the corresponding data partition.

[0064] like Figure 4The data flow diagram of a data table synchronization method based on the Spark framework in one embodiment of the present application is shown. Among them, the number of metadata of the data table to be processed in the MySQL database is 100 in total, but the range of the primary key temporary table formed by the primary key corresponding to the metadata is 1-200. According to the above steps, 1 is used as the initial primary key to generate 1-100 auto-increment fields corresponding to each primary key in the primary key temporary table as an auto-increment field table. And based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed, the average processable data volume of each data partition is 20. Further, at least one auto-increment field and the average processable data volume are modulo-operated, and the auto-increment field table is divided at the division point where the remainder is equal to 0. The auto-increment field table can be evenly distributed into 5 partitions [1-20], [21-40], [41-60], [61-80] and [81-100].

[0065] Furthermore, the primary key in the corresponding primary key temporary table is found through the auto-increment field. For example, auto-increment field 1 corresponds to primary key 2, auto-increment field 20 corresponds to primary key 30, etc. The primary key can be divided into 5 partitions with the same number of entries (all 20 entries) [2-30], [35-80], [111-135], [140-171], [175-200]. And, based on the primary key value, the corresponding metadata is further obtained and concurrently synchronized to the corresponding data partitions 1-5.

[0066] The data table synchronization method based on the Spark framework in the embodiment of the present application is consistent with the aforementioned execution steps when the primary key is a non-integer field. For example, if the upper and lower limits of the non-integer primary key pulled by the Spark framework are AAA-FFF, and the total number of data items is 40, according to the aforementioned steps, 1 is used as the initial primary key to generate an auto-increment field table of 1-40 corresponding to each primary key in the primary key temporary table. Figure 4 Similar to the method in, the metadata is synchronized with the corresponding data partition.

[0067] The embodiment of the present application is based on the Spark framework to synchronize the data table to be processed in the MySQL database with uneven primary keys or non-integer primary keys, and adds the step of calculating the custom partition interval, which solves the data skew problem caused by uneven primary key distribution during synchronization and reduces the generation of temporary files. It helps to solve the data skew problem caused by the Spark framework pulling the data table with uneven primary key distribution and the problem of generating a large number of temporary files in the MySQL node when the Spark framework pulls the data table of the MySQL database with large data volume and non-integer primary key, and realizes the universal custom partitioning function of the data table in the MySQL database with non-integer primary key and uneven primary key distribution in the Spark framework synchronization under the background of large task volume.

[0068] Based on the same inventive concept as the aforementioned data table synchronization method based on the Spark framework, an embodiment of the present application also provides a data table synchronization device based on the Spark framework.

[0069] Figure 5 This is a schematic diagram of a data table synchronization device based on the Spark framework in another embodiment of the present application. The synchronization device may include:

[0070] An acquisition module 51 is used to acquire a primary key of at least one metadata in a to-be-processed data table based on a Spark framework, and generate an auto-increment field table of auto-increment fields corresponding to at least one primary key in the to-be-processed data table based on the acquired number of primary keys;

[0071] A calculation module 52, configured to calculate an average processable data volume of each data partition based on the data processing capability of each data partition and the average length of each metadata in the data table to be processed;

[0072] The synchronization module 53 is used to divide the auto-increment field table according to the auto-increment field table and the average processable data volume, and divide the data partition where at least one primary key is located based on the auto-increment field contained in at least one sub-auto-increment field table after the division, and synchronize the metadata corresponding to the primary key in the data table to be processed after the division to the corresponding data partition.

[0073] In this embodiment, the specific functions and interaction modes of the acquisition module 51, the calculation module 52 and the synchronization module 53 can be found in Figure 1 and Figure 2 The description of the corresponding embodiments will not be repeated here.

[0074] Optionally, the acquisition module 51 is further used for:

[0075] Register the obtained primary key into the primary key temporary table;

[0076] Starting with the initial primary key in the primary key temporary table, and based on the number of primary keys obtained, a corresponding auto-increment field is generated for each primary key in the temporary table, and an auto-increment field table corresponding to the primary key temporary table is generated.

[0077] Optionally, the synchronization module 53 is further used for:

[0078] Traverse at least one auto-increment field in the auto-increment field table, perform a modulo operation on at least one auto-increment field and the average processable data volume, and when there is a remainder and the remainder is 0, use the corresponding auto-increment field as a division point, and divide the auto-increment field table into sub-auto-increment field tables with the same number as the number of division points.

[0079] Optionally, the synchronization module 53 is further used for:

[0080] Match the corresponding data partition for each sub-auto-increment field table, and divide the primary key corresponding to the auto-increment field in each sub-auto-increment field into the corresponding data partition;

[0081] Based on the primary key corresponding to each data partition, the metadata in the to-be-processed data table corresponding to the primary key is concurrently pulled, and the metadata is synchronized to the corresponding data partition.

[0082] Figure 6 A schematic diagram of an electronic device provided by another embodiment of the present application. Figure 6 As shown, another embodiment of the present application further provides an electronic device, which may include a processor 601, wherein the processor 601 is used to execute the steps of the above-mentioned method for synchronizing a data table based on a Spark framework. Figure 6 It can also be seen that the electronic device provided in the above embodiment also includes a non-transitory computer-readable storage medium 602, on which a computer program is stored. When the computer program is executed by the processor 601, the steps of the above-mentioned method for synchronizing a data table based on the Spark framework are executed.

[0083] Specifically, the non-transitory computer-readable storage medium 602 can be a general storage medium, such as a mobile disk, a hard disk, FLASH, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), or a portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 602 is executed by the processor 601, it can trigger the processor 601 to execute the above-mentioned steps in a method for synchronizing a data table based on the Spark framework.

[0084] In practical applications, the non-transitory computer-readable storage medium 602 may be included in the device / apparatus / system described in the above embodiment, or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, each step in the above method for synchronizing a data table based on the Spark framework can be executed.

[0085] The embodiment of the present invention also provides a computer-readable medium, the computer-readable storage medium stores instructions, and the instructions can execute the steps in the data table synchronization method based on the Spark framework as described above when executed by a processor. The computer-readable medium in the actual application may be included in the device / apparatus / system described in the above embodiment, or it may exist alone without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data table synchronization method based on the Spark framework described in the above embodiments can be implemented. In the embodiment disclosed in the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device. Moreover, the embodiment of the present invention also provides a computer program product including computer instructions. When the computer instructions are executed by the processor, the data table synchronization method based on the Spark framework described in the above embodiment is implemented.

[0086] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order marked in different drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0088] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still change the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these changes, modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data table synchronization method based on Spark framework, characterized in that: include: Acquire at least one primary key of metadata in the data table to be processed based on the Spark framework, and generate an auto-increment field table of auto-increment fields corresponding to at least one primary key in the data table to be processed based on the number of acquired primary keys; Based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed, the average processable data volume of the data partition is calculated; The auto-increment field table is divided according to the auto-increment field table and the average processable data volume, and at least one data partition where the primary key is located is divided according to the auto-increment field contained in at least one sub-auto-increment field table after the division, and each metadata corresponding to the primary key in the to-be-processed data table after the division is synchronized to the corresponding data partition; The method of dividing the data partition where at least one primary key is located based on the auto-increment field contained in at least one auto-increment field table after the division, and synchronizing each metadata corresponding to the primary key in the to-be-processed data table after the division to the corresponding data partition includes: Match the corresponding data partition for each sub-auto-increment field table, and divide the primary key corresponding to the auto-increment field in each sub-auto-increment field into the corresponding data partition; Based on the primary key corresponding to each data partition, the metadata in the to-be-processed data table corresponding to the primary key is concurrently pulled, and the metadata is synchronized to the corresponding data partition.

2. The synchronization method according to claim 1, characterized in that: The step of generating an auto-increment field table of the auto-increment field corresponding to at least one primary key in the data table to be processed based on the acquired number of primary keys includes: Register the obtained primary key into the primary key temporary table; Starting with the initial primary key in the primary key temporary table, and based on the number of primary keys obtained, a corresponding auto-increment field is generated for each primary key in the temporary table, and an auto-increment field table corresponding to the primary key temporary table is generated.

3. The synchronization method according to claim 1, characterized in that: The steps of dividing the auto-increment field table according to the average processable data volume include: Traverse at least one auto-increment field in the auto-increment field table, perform a modulo operation on at least one auto-increment field and the average processable data volume, and when there is a remainder and the remainder is 0, use the corresponding auto-increment field as a division point, and divide the auto-increment field table into sub-auto-increment field tables with the same number as the number of division points.

4. A data table synchronization device based on Spark framework, characterized in that: include: An acquisition module is used to acquire a primary key of at least one metadata in a to-be-processed data table based on a Spark framework, and generate an auto-increment field table of auto-increment fields corresponding to at least one primary key in the to-be-processed data table based on the acquired number of primary keys; A calculation module, used to calculate an average processable data volume of each data partition based on the data processing capacity of each data partition and the average length of each metadata in the data table to be processed; A synchronization module is used to divide the auto-increment field table according to the auto-increment field table and the average processable data volume, and divide the data partition where at least one primary key is located based on the auto-increment field contained in at least one sub-auto-increment field table after the division, and synchronize each metadata corresponding to the primary key in the to-be-processed data table after the division to the corresponding data partition; Wherein, the synchronization module is also used for: Match the corresponding data partition for each sub-auto-increment field table, and divide the primary key corresponding to the auto-increment field in each sub-auto-increment field into the corresponding data partition; Based on the primary key corresponding to each data partition, the metadata in the to-be-processed data table corresponding to the primary key is concurrently pulled, and the metadata is synchronized to the corresponding data partition.

5. The synchronization device according to claim 4, characterized in that: The Get module is also used to: Register the obtained primary key into the primary key temporary table; Starting with the initial primary key in the primary key temporary table, and based on the number of primary keys obtained, a corresponding auto-increment field is generated for each primary key in the temporary table, and an auto-increment field table corresponding to the primary key temporary table is generated.

6. The synchronization device according to claim 4, characterized in that: The sync module is also used to: Traverse at least one auto-increment field in the auto-increment field table, perform a modulo operation on at least one auto-increment field and the average processable data volume, and when there is a remainder and the remainder is 0, use the corresponding auto-increment field as a division point, and divide the auto-increment field table into sub-auto-increment field tables with the same number as the number of division points.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions which, when executed by a processor, cause the processor to perform the synchronization method according to any one of claims 1 to 3.

8. An electronic device, characterized in that: The electronic device includes: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the synchronization method described in any one of claims 1 to 3.

9. A computer program product, characterized in that The method comprises computer instructions, which implement the synchronization method according to any one of claims 1 to 3 when executed by a processor.

Images