Method, device, apparatus and computer-readable medium for processing data

Abstract

The present invention discloses a method, device, equipment and computer-readable medium for processing data, and relates to the field of computer technology. A specific implementation of the method includes: according to the table partition in the data source, deleting the data of the corresponding table partition in the version table, wherein the table partition is preset according to business requirements; pushing the data of the table partition in the data source to the corresponding table partition in the version table, wherein the format of the version table is consistent with the format of the formal table; after successfully verifying the data in the corresponding table partition in the version table, copying the data in the corresponding table partition in the version table to the formal table. This implementation can shorten the window period of the data push process and reduce data fluctuations.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and computer-readable medium for processing data. Background Art

[0002] Data push is an important part of the data chain. The commonly used data push method is to first obtain the required data from the data source, and then import the data according to the partition field. If there is already data under the partition, delete the data under the partition first and then push the data.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: because the data needs to be deleted first and then pushed for a period of time, there are window periods and drastic fluctuations in the data. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device and computer-readable medium for processing data, which can shorten the window period of the data push process and reduce data fluctuations.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for processing data is provided, comprising:

[0006] According to the table partition in the data source, delete the data of the corresponding table partition in the version table, where the table partition is preset according to business requirements;

[0007] Push the data of the table partition in the data source to the corresponding table partition in the version table, where the format of the version table is consistent with that of the formal table;

[0008] After successfully verifying the data in the corresponding table partition in the version table, copy the data in the corresponding table partition in the version table to the formal table.

[0009] The data in the corresponding table partition in the version table that is successfully verified includes:

[0010] Check that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source;

[0011] Confirm that the business indicators of the data in the corresponding table partition in the version table are within the business threshold range.

[0012] The checking that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source includes:

[0013] If the data volume of the corresponding table partition in the version table is different from the data volume of the table partition in the data source, the data volume of each shard in the corresponding table partition in the version table is determined to obtain shards with different shard data volumes;

[0014] Based on the shards with different data volumes, data is pushed again from the shards in the table partition in the data source to the version table until it is checked that the data volume of each shard in the corresponding table partition in the version table is the same as the data volume of each shard in the table partition in the data source.

[0015] Determining the data volume of each shard in the corresponding table partition in the version table includes:

[0016] Hash processing is performed in sequence on the high discreteness fields and index fields of each shard in the version table to determine the data volume of each shard in the corresponding table partition in the version table.

[0017] The step of copying the data in the corresponding table partition in the version table to the formal table includes:

[0018] Checking that a partition in the formal table corresponding to a table partition in the data source does not store data;

[0019] Copy the data in the corresponding table partition in the version table to the formal table.

[0020] The step of copying the data in the corresponding table partition in the version table to the formal table includes:

[0021] Checking that a partition in the formal table corresponding to a table partition in the data source has stored data;

[0022] Storing the stored data in the cache table and deleting the stored data;

[0023] Copy the data in the corresponding table partition in the version table to the formal table.

[0024] After copying the data in the corresponding table partition in the version table to the formal table, the method further includes:

[0025] Check that the amount of data in each shard in the corresponding table partition in the formal table is the same as the amount of data in each shard in the table partition in the data source.

[0026] According to a second aspect of an embodiment of the present invention, there is provided a device for processing data, including:

[0027] A deletion module is used to delete data of corresponding table partitions in the version table according to table partitions in the data source, where the table partitions are preset according to business requirements;

[0028] A push module, used to push the data of the table partition in the data source to the corresponding table partition in the version table, wherein the format of the version table is consistent with the format of the formal table;

[0029] The processing module is used to copy the data in the corresponding table partition in the version table to the formal table after successfully verifying the data in the corresponding table partition in the version table.

[0030] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for processing data, including:

[0031] one or more processors;

[0032] a storage device for storing one or more programs,

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0034] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method as described above is implemented.

[0035] One embodiment of the above invention has the following advantages or beneficial effects: according to the table partition in the data source, the data of the corresponding table partition in the version table is deleted, and the table partition is preset according to business needs; the data of the table partition in the data source is pushed to the corresponding table partition in the version table, and the format of the version table is consistent with the format of the formal table; after successfully verifying the data in the corresponding table partition in the version table, the data in the corresponding table partition in the version table is copied to the formal table. Through the version table, the data in the data source is copied to the formal table, thereby shortening the window period of the data push process and reducing data fluctuations.

[0036] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0038] Figure 1 is a schematic diagram of the main flow of a method for processing data according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a process for verifying data in a corresponding table partition in a successful version table according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a process for checking the same amount of data according to an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of a process of replicating data in a corresponding table partition in a version table to a formal table according to an embodiment of the present invention;

[0042] Figure 5 is another schematic diagram of a process of replicating data in a corresponding table partition in a version table to a formal table according to an embodiment of the present invention;

[0043] Figure 6 is a schematic diagram of the process of Clickhouse processing data according to an embodiment of the present invention;

[0044] Figure 7 is a schematic diagram of the main structure of a device for processing data according to an embodiment of the present invention;

[0045] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0046] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0048] At present, the commonly used data push method is to first obtain the required data from the data source. As a real-time analytical database, Clickhouse itself bears more query functions, which requires business data to be processed and stored in databases such as MySQL, Hive, and Kafka, and then the result data is transferred to Clickhouse. Therefore, data push is an important part of the data link.

[0049] Then, import data into Clickhouse according to the partition field in Clickhouse. If there is already data under the partition, delete the data under the partition first and then push the data.

[0050] However, data push results in data gaps and sharp fluctuations.

[0051] In order to solve the problems of data window periods and sharp fluctuations, the following technical solutions in the embodiments of the present invention may be adopted.

[0052] See also Figure 1 , Figure 1 Schematic diagram of the main process of the method for processing data according to an embodiment of the present invention, in which the data in the data source is pushed to the formal table through the version table. Figure 1 As shown, the specific steps include:

[0053] S101. According to the table partition in the data source, delete the data of the corresponding table partition in the version table. The table partition is preset according to business requirements.

[0054] In an embodiment of the present invention, a version table is added during the data source push process, and then the data in the data source is pushed to the formal table through the version table. The format of the version table is consistent with the format of the formal table to ensure that the data in the version table can be directly pushed to the formal table for storage. As an example, the formal table includes the Clickhouse formal table.

[0055] As for the data source, the data source may be one of the following: Hive, MYSQL and KAFKA. Various data sources are applicable to the technical solution in the embodiment of the present invention.

[0056] The data source includes multiple table partitions, which are preset according to business requirements. As an example, the data involved in business requirement 1 is stored in table partition A1, and the data involved in business requirement 2 is stored in table partition A2. Setting table partitions in the data source can improve the efficiency of data processing.

[0057] The function of the version table is to transfer data from the data source to the formal table, so the version table includes multiple table partitions. The table partitions in the version table correspond to the table partitions in the data source. As an example, the table partitions in the data source include A1, A2, and A3, and the table partitions in the version table include B1, B2, and B3. Among them, A1 corresponds to B1, A2 corresponds to B2, and A3 corresponds to B3.

[0058] In order to push the table partition data in the data source to the corresponding table partition in the version table to improve the success rate of the push, it is necessary to delete the data of the corresponding table partition in the version table. Since the data in the version table is often used to push to the formal table, after the data in the version table has been pushed to the formal table, deleting the data of the corresponding table partition in the version table will not affect the processing data.

[0059] As an example, the data source is Hive, and the default table partition is dt. Based on dt in Hive, delete the data in the partition corresponding to dt in the version table.

[0060] S102: Push the data of the table partition in the data source to the corresponding table partition in the version table. The format of the version table is consistent with the format of the formal table.

[0061] After deleting the data of the corresponding table partition in the version table, you can push the data of the table partition in the data source to the corresponding table partition in the version table. The format of the version table is consistent with that of the formal table to ensure that the data of the version table can be directly pushed to the formal table for storage.

[0062] As an example, the data source is Hive, and the default table partition is dt. After deleting the data in the partition corresponding to dt in the version table, push the data of dt in Hive directly to the partition corresponding to dt in the version table.

[0063] S103. After successfully verifying the data in the corresponding table partition in the version table, copy the data in the corresponding table partition in the version table to the formal table.

[0064] The prerequisite for copying the data in the corresponding table partition in the version table to the formal table is to verify the data in the corresponding table partition in the version table successfully. In other words, only when the data in the corresponding table partition in the version table is consistent with the data in the table partition in the data source, the data in the version table will be copied to the formal table to improve the accuracy of the formal table data.

[0065] See also Figure 2 , Figure 2 1 is a schematic diagram of a process for verifying data in a corresponding table partition in a successful version table according to an embodiment of the present invention, specifically comprising the following steps:

[0066] S201. Check that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source.

[0067] In the embodiment of the present invention, the version table is detected by the data volume. Specifically, it is detected whether the data volume pushed into the version table is equal to the output data volume of the data source.

[0068] If the amount of data pushed into the version table is equal to the amount of data output from the data source, then the amount of data in each shard in the corresponding table partition in the version table is determined to be the same as the amount of data in each shard in the table partition in the data source.

[0069] If the amount of data pushed into the version table is not equal to the amount of data output from the data source, the reason is that too much data has been pushed or data has been missed. In order to ensure the accuracy of the data, it is necessary to push the data again to maintain data consistency.

[0070] In the existing technology, the output data in the data source is often re-pushed to the version table. For large amounts of data, the time cost of re-pushing is very high, and there is a high probability that multiple pushes or missed pushes will occur again, thereby prolonging the push process.

[0071] It has been learned through many practices that when there are over-pushing or missing pushes during the push process, it is likely that some of the shard data are wrong due to system reasons. Therefore, in the embodiment of the present invention, the data is rolled back in shard units to push again, which can greatly shorten the entire push cycle compared to re-pushing the entire data.

[0072] See also Figure 3 , Figure 3 1 is a flow chart of checking the same amount of data according to an embodiment of the present invention, which specifically includes the following steps:

[0073] S301. If the data volume of the corresponding table partition in the version table is different from the data volume of the table partition in the data source, determine the data volume of each shard in the corresponding table partition in the version table to obtain shards with different shard data volumes.

[0074] First, the data in the corresponding table partition in the version table is sharded, that is, the above data is divided into multiple pieces. The number of shards is pre-set based on the distributed machine nodes. Then, the data volume of each shard is checked in units of shards.

[0075] For each shard's high-discreteness field and index field in the version table, hash processing is performed in turn to determine the amount of data in each shard in the corresponding table partition in the version table. Among them, the high-discreteness field refers to the data obtained by evenly splitting the data on the same shard.

[0076] Highly discrete fields are evenly distributed on each shard to avoid data skew. Data skew means that the cached data is not dispersed enough, resulting in a large amount of cached data being concentrated on one or several service nodes.

[0077] As an example, the official table is the Clickhouse official table. When pushing data to Clickhouse, the data is divided according to the number of shards. Then, HASH processing is performed according to Formula 1 to obtain the data volume of the shards.

[0078] COALCASE(CAST(ABS(HASH(A, B, high discrete field))as int)%Number of shards, 0) Formula 1

[0079] Among them, A and B are index fields. When the discreteness of the index field is not high, add one or more fields with high discreteness. HASH() is a function for calculating hash values. ABS() is a function for calculating the absolute value of hash values. CAST() is to convert the value of the absolute value function into an int type. The function of COALCASE() is: if the result of CAST() is null, the result 0 is returned.

[0080] Determine the data volume of each shard in the corresponding table partition in the version table, and determine the data volume of each shard in the table partition in the data source. By comparing the data volume of each shard in the version table and the data source, the shards with different data volumes are known.

[0081] S302: Based on shards with different shard data volumes, push data from the shards in the table partition in the data source to the version table again until it is checked that the data volume of each shard in the corresponding table partition in the version table is the same as the data volume of each shard in the table partition in the data source.

[0082] After locating the shard with different data volume in the corresponding table partition in the version table, you can push the data of the shard from the data source instead of all the data. That is, push the data from the shard in the table partition in the data source to the version table again. After pushing the shard data, check the data volume of the shard again. If the data volume of the shard in the corresponding table partition in the version table is the same as the data volume of the shard in the table partition in the data source, the check ends; otherwise, you need to push the data of the shard again.

[0083] exist Figure 3 In the embodiment, the shards with different data volumes are located, and the data to be pushed to the shards is re-pushed. For data tables with large data volumes and prone to push problems, the success rate of push can be greatly improved and the time loss of re-pushing can be reduced.

[0084] S202: Confirm that the business indicators of the data in the corresponding table partition in the version table are within the business threshold range.

[0085] After checking that the data volume of each shard is the same, you need to confirm whether the business indicator is within the business threshold range. Specifically, select the business indicator and then check it according to the business threshold range.

[0086] As an example, the business indicator includes page views (PV). Perform a sum() calculation on the PV of the data in the corresponding table partition in the version table, and use the data in the previous time period as the business threshold range. If the calculated value is not within the business threshold range, the data is abnormal. There is no need to copy the data in the version table to the formal table; if the calculated value is within the business threshold range, the data is normal, and the data in the version table is copied to the formal table.

[0087] exist Figure 2 In the embodiment, the data volume of each shard and the business indicators are checked to verify the data in the version table, thereby ensuring the accuracy of the data in the version table.

[0088] After successfully verifying the data in the corresponding table partition in the version table, you can copy the data in the corresponding table partition in the version table to the formal table.

[0089] See also Figure 4 , Figure 4 This is a schematic diagram of a process of replicating data in a corresponding table partition in a version table to a formal table according to an embodiment of the present invention, and specifically includes the following steps:

[0090] S401. Check the partition in the formal table that corresponds to the table partition in the data source and find that no data is stored.

[0091] After successfully verifying the data in the corresponding table partition in the version table, you need to check whether the partition in the formal table that corresponds to the table partition in the data source stores data to determine whether to directly copy the data in the version table to the formal table.

[0092] The partition in the formal table corresponding to the table partition in the data source is checked. If no data is stored, S402 may be executed.

[0093] S402. Copy the data in the corresponding table partition in the version table to the formal table.

[0094] After the check is completed, the data in the corresponding table partition in the version table is copied to the formal table, and the corresponding sub-partition is added to the formal table. The copy can be achieved through the ATTACH method.

[0095] The above replication process is a process from local table to local table, rather than a distributed table. This can reduce network overhead and further shorten the window period for updating formal table data. As an example, the ATTACH method includes:

[0096] ALTER TABLE A_local ATTACH PARTITION(dt)FROM A_tmp_local.

[0097] exist Figure 4 In the embodiment, by replicating the version table to the formal table, the network overhead is reduced, thereby shortening the window period for updating the formal table data.

[0098] See also Figure 5 , Figure 5 This is another schematic diagram of a process of replicating data in a corresponding table partition in a version table to a formal table according to an embodiment of the present invention, and specifically includes the following steps:

[0099] S501. Check the partition in the formal table that corresponds to the table partition in the data source and whether data has been stored therein.

[0100] Check the partitions in the formal table that correspond to the table partitions in the data source. If data has been stored, you cannot directly copy the data in the version table to the formal table.

[0101] S502: Store the stored data in the cache table, and delete the stored data.

[0102] If the partition in the formal table corresponding to the table partition in the data source has stored data, the data in the partition corresponding to the formal table can be transferred to the cache table, and then the data in the partition corresponding to the formal table can be deleted.

[0103] As an example, the data in the partition corresponding to the formal table is transferred to the cache table through the ATTACH method. The purpose of this is to back up the original data in the formal table. When the formal table data is abnormal, data rollback can be effectively achieved.

[0104] In one embodiment of the present invention, if an exception occurs in the formal table, data rollback is performed according to the data in the cache table.

[0105] After the formal table backup is completed, perform a DROP operation on the corresponding partition in the formal table, and then check whether the data under the partition in each node is completely deleted to prevent the undeleted data from affecting the new data.

[0106] S503. Copy the data in the corresponding table partition in the version table to the formal table.

[0107] Similar to S402, after the check is completed, the data in the corresponding table partition in the version table is copied to the formal table, and the corresponding sub-partition is added to the formal table. The copying can be implemented by ATTACH method.

[0108] exist Figure 5 In the embodiment of the present invention, the cache table is used to store the existing data in the formal table so as to implement data rollback. The network overhead is also reduced by copying the version table to the formal table, thereby shortening the window period for updating the formal table data.

[0109] Finally, check the data volume of the corresponding table partition in the official table and the data volume of the table partition in the data source. If the data volume of the two partitions is the same, complete the data push from the data source to the official table; if the data volume of the two partitions is different, first roll back the data through the cache table, and then exit the data push process.

[0110] In one embodiment of the present invention, in order to verify the data volume and avoid re-pushing data from the data source to the formal table, the data volume of the corresponding table partition in the formal table and the data volume of the table partition in the data source can be checked according to the shards.

[0111] Specifically, check that the amount of data in each shard in the corresponding table partition in the formal table is the same as the amount of data in each shard in the table partition in the data source. Figure 3 The technical solution in the embodiment.

[0112] In the above-mentioned embodiment of the present invention, according to the table partition in the data source, the data of the corresponding table partition in the version table is deleted, and the table partition is preset according to business needs; the data of the table partition in the data source is pushed to the corresponding table partition in the version table, and the format of the version table is consistent with the format of the formal table; after successfully verifying the data in the corresponding table partition in the version table, the data in the corresponding table partition in the version table is copied to the formal table. Through the version table, the data in the data source is copied to the formal table, thereby shortening the window period of the data push process and reducing data fluctuations.

[0113] The embodiment of the present invention, on the one hand, avoids long-term push data updates to normal tables by establishing a version table, greatly reducing the user's perception of the push data process. On the other hand, sharded data verification is adopted, so that data rollback is implemented in units of shards, thereby avoiding data fluctuations caused by full data rollback.

[0114] See also Figure 6 , Figure 6 It is a schematic diagram of the process of Clickhouse processing data according to an embodiment of the present invention.

[0115] Figure 6 The data source in is Hive, and the default table partition is dt. In the process of pushing data from Hive to the official Clickhouse table, the process of pushing data to the version table is added.

[0116] First, according to dt in Hive, delete the data of the corresponding dt partition in the version table. Then, extract the data in Hive and push it directly to the version table. After the corresponding partition data in the version table passes the data volume verification and business indicator verification, the data is pushed to the Clickhouse official table.

[0117] The data volume verification is performed on a shard basis. If the shard data verification fails, the shard data is rolled back, thereby greatly shortening the push cycle.

[0118] In addition, a cache table is set up to store the original data of the formal table. When the formal table data is abnormal, data rollback can be performed to restore the formal table.

[0119] See also Figure 7 , Figure 7 is a schematic diagram of the main structure of a data processing device according to an embodiment of the present invention. The data processing device can implement a method for processing data, such as Figure 7 As shown, the device for processing data specifically includes:

[0120] A deletion module 701 is used to delete data of a corresponding table partition in the version table according to the table partition in the data source, where the table partition is preset according to business requirements;

[0121] A push module 702 is used to push the data of the table partition in the data source to the corresponding table partition in the version table, and the format of the version table is consistent with the format of the formal table;

[0122] The processing module 703 is used to copy the data in the corresponding table partition in the version table to the formal table after successfully verifying the data in the corresponding table partition in the version table.

[0123] In one embodiment of the present invention, the processing module 703 is specifically used to check that the amount of data of each shard in the corresponding table partition in the version table is the same as the amount of data of each shard in the table partition in the data source;

[0124] Confirm that the business indicators of the data in the corresponding table partition in the version table are within the business threshold range.

[0125] In one embodiment of the present invention, the processing module 703 is specifically used to determine the data volume of each shard in the corresponding table partition in the version table if the data volume of the corresponding table partition in the version table is different from the data volume of the table partition in the data source to obtain the shards with different shard data volumes;

[0126] Based on the shards with different data volumes, data is pushed again from the shards in the table partition in the data source to the version table until it is checked that the data volume of each shard in the corresponding table partition in the version table is the same as the data volume of each shard in the table partition in the data source.

[0127] In one embodiment of the present invention, the processing module 703 is specifically used to perform hash processing on the high discreteness fields and index fields of each shard in the version table in sequence to determine the data volume of each shard in the corresponding table partition in the version table.

[0128] In one embodiment of the present invention, the processing module 703 is specifically used to check if a partition in the formal table corresponding to the table partition in the data source stores no data;

[0129] Copy the data in the corresponding table partition in the version table to the formal table.

[0130] In one embodiment of the present invention, the processing module 703 is specifically used to check the partition in the formal table corresponding to the table partition in the data source, and has stored data;

[0131] Storing the stored data in the cache table and deleting the stored data;

[0132] Copy the data in the corresponding table partition in the version table to the formal table.

[0133] In one embodiment of the present invention, the processing module 703 is further used to check that the data volume of each shard in the corresponding table partition in the formal table is the same as the data volume of each shard in the table partition in the data source.

[0134] Figure 8 An exemplary system architecture 800 is shown to which the method for processing data or the apparatus for processing data according to the embodiments of the present invention can be applied.

[0135] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0136] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0137] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0138] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0139] It should be noted that the method for processing data provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the device for processing data is generally set in the server 805 .

[0140] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0141] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0143] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0144] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0145] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a deletion module, a push module, and a processing module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the deletion module may also be described as "used to delete data corresponding to table partitions in the version table according to table partitions in the data source, wherein the table partitions are preset according to business requirements."

[0148] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0149] According to the table partition in the data source, delete the data of the corresponding table partition in the version table, where the table partition is preset according to business requirements;

[0150] Push the data of the table partition in the data source to the corresponding table partition in the version table, where the format of the version table is consistent with that of the formal table;

[0151] After successfully verifying the data in the corresponding table partition in the version table, copy the data in the corresponding table partition in the version table to the formal table.

[0152] According to the technical solution of the embodiment of the present invention, according to the table partition in the data source, the data of the corresponding table partition in the version table is deleted, and the table partition is preset according to business needs; the data of the table partition in the data source is pushed to the corresponding table partition in the version table, and the format of the version table is consistent with the format of the formal table; after successfully verifying the data in the corresponding table partition in the version table, the data in the corresponding table partition in the version table is copied to the formal table. Through the version table, the data in the data source is copied to the formal table, thereby shortening the window period of the data push process and reducing data fluctuations.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing data, characterized in that: include: According to the table partition in the data source, delete the data of the corresponding table partition in the version table, where the table partition is preset according to business requirements; Push the data of the table partition in the data source to the corresponding table partition in the version table, where the format of the version table is consistent with that of the formal table; After successfully verifying the data in the corresponding table partition in the version table, copy the data in the corresponding table partition in the version table to the formal table; The data in the corresponding table partition in the version table that is successfully verified includes: Check that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source; Confirm that the business indicators of the data in the corresponding table partition in the version table are within the business threshold range; The checking that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source includes: If the data volume of the corresponding table partition in the version table is different from the data volume of the table partition in the data source, the data volume of each shard in the corresponding table partition in the version table is determined to obtain shards with different shard data volumes; Based on the shards with different data volumes, data is pushed again from the shards in the table partition in the data source to the version table until it is checked that the data volume of each shard in the corresponding table partition in the version table is the same as the data volume of each shard in the table partition in the data source.

2. The method for processing data according to claim 1, characterized in that: Determining the data volume of each shard in the corresponding table partition in the version table includes: Hash processing is performed in sequence on the high discreteness fields and index fields of each shard in the version table to determine the data volume of each shard in the corresponding table partition in the version table.

3. The method for processing data according to claim 1, characterized in that: The step of copying the data in the corresponding table partition in the version table to the formal table includes: Checking that a partition in the formal table corresponding to a table partition in the data source does not store data; Copy the data in the corresponding table partition in the version table to the formal table.

4. The method for processing data according to claim 1, characterized in that: The step of copying the data in the corresponding table partition in the version table to the formal table includes: Checking that a partition in the formal table corresponding to a table partition in the data source has stored data; Storing the stored data in the cache table and deleting the stored data; Copy the data in the corresponding table partition in the version table to the formal table.

5. The method for processing data according to claim 4, characterized in that: After copying the data in the corresponding table partition in the version table to the formal table, the method further includes: Check that the amount of data in each shard in the corresponding table partition in the formal table is the same as the amount of data in each shard in the table partition in the data source.

6. A device for processing data, characterized in that: include: A deletion module is used to delete data of corresponding table partitions in the version table according to table partitions in the data source, where the table partitions are preset according to business requirements; A push module, used to push the data of the table partition in the data source to the corresponding table partition in the version table, wherein the format of the version table is consistent with the format of the formal table; A processing module, configured to copy the data in the corresponding table partition in the version table to the formal table after successfully verifying the data in the corresponding table partition in the version table; and, checking that the amount of data in each shard in the corresponding table partition in the version table is the same as the amount of data in each shard in the table partition in the data source; Confirm that the business indicators of the data in the corresponding table partition in the version table are within the business threshold range; And, if the data volume of the corresponding table partition in the version table is different from the data volume of the table partition in the data source, the data volume of each shard in the corresponding table partition in the version table is determined to obtain shards with different shard data volumes; Based on the shards with different data volumes, data is pushed again from the shards in the table partition in the data source to the version table until it is checked that the data volume of each shard in the corresponding table partition in the version table is the same as the data volume of each shard in the table partition in the data source.

7. An electronic device for processing data, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images