Data updating method and device
By creating temporary data files and replacing file names during data update, the problem of insufficient data stability during data update is solved, ensuring the continuity and stability of data queries.
Patent Information
- Application Number
- CN202110584139.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-27
- Publication Date
- 2025-03-18
- Estimated Expiration
- 2041-05-27
AI Technical Summary
During data update process, the prior art leads to a lack of stability in data, especially during historical data deletion, which may lead to data query failure or error.
By creating a temporary data file to store a new full amount of data, and after the data storage is completed, the file names of the temporary file and the official file are replaced, and then the historical data is deleted, thereby ensuring the continuity of data queries.
It realizes that data query requests can always be processed during data update process, improving data stability during data update process.
Smart Images
Figure CN113326232B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to computer technology, and more particularly to a data updating method and device. Background Art
[0002] The system usually involves data update operations. There are usually two ways to update data: one is full update and the other is incremental update.
[0003] In the current related technology, when implementing a full update of data, all historical data in the system is usually deleted, and then the full data is added to the system to achieve a full update of the data. However, if there is a data query request during the process of deleting historical data, it will cause data query failure or query error.
[0004] Therefore, the current data update method based on full update will lead to a lack of data stability during the data update process. Summary of the invention
[0005] The embodiments of the present application provide a data updating method and device to overcome the problem of lack of data stability during the data updating process.
[0006] In a first aspect, an embodiment of the present application provides a data updating method, comprising:
[0007] Obtain new full data of the first business;
[0008] Determine a first data file corresponding to the first service, the first data file including historical data of the first service, the file name of the first data file being a first name corresponding to the first service, the first name being used to access data corresponding to the first service;
[0009] Creating a second data file, and storing the new full data in the second data file;
[0010] The file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
[0011] In a possible design, creating the second data file includes:
[0012] Determining a first file path, the first file path being different from a file path of the first data file;
[0013] The second data file is created according to the first file path.
[0014] In a possible design, determining the first file path includes:
[0015] Obtain the current time and a preset file path set, wherein the preset file path set includes the first file path;
[0016] According to the current time, the first file path is determined in the preset file path set.
[0017] In a possible design, the current time is the current date, and the preset file path set includes a first path and a second path;
[0018] The determining the first file path in the preset file path set according to the current time includes:
[0019] Determine the difference between the current date and the preset date as the date difference;
[0020] If the date difference is an odd number, determining the first path in the preset file path set as the first file path;
[0021] If the date difference is an even number, the second path in the preset file path set is determined as the first file path.
[0022] In a possible design, updating the file name of the first data file to a name other than the first name, and updating the file name of the second data file to the first name, includes:
[0023] Determining a preset data volume of the new full amount of data;
[0024] If the data amount in the second data file is equal to the preset data amount, the file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
[0025] In one possible design, the method further includes:
[0026] Deleting historical data in the first data file;
[0027] Delete the first data file.
[0028] In one possible design, the historical data in the first data file is stored in at least one node;
[0029] Deleting the historical data in the first data file includes:
[0030] Each of the nodes is traversed in sequence, and the historical data in each of the nodes is deleted in sequence.
[0031] In a possible design, each of the nodes includes at least one partition, and the historical data in the first data file is stored in each of the partitions of each of the nodes;
[0032] The deleting the historical data in each of the nodes in sequence includes:
[0033] For any of the nodes, the partitions in the node are traversed in sequence, and the historical data in the partitions are deleted in sequence.
[0034] In a second aspect, an embodiment of the present application provides a data updating device, including:
[0035] An acquisition module, used to acquire new full data of the first service;
[0036] A determination module, configured to determine a first data file corresponding to the first service, wherein the first data file includes historical data of the first service, and the file name of the first data file is a first name corresponding to the first service, and the first name is used to access data corresponding to the first service;
[0037] A creation module, used for creating a second data file, and storing the new full data in the second data file;
[0038] An updating module is used to update the file name of the first data file to a name other than the first name, and to update the file name of the second data file to the first name.
[0039] In a possible design, the creation module is specifically used to:
[0040] Determining a first file path, the first file path being different from a file path of the first data file;
[0041] The second data file is created according to the first file path.
[0042] In a possible design, the creation module is specifically used to:
[0043] Obtain the current time and a preset file path set, wherein the preset file path set includes the first file path;
[0044] According to the current time, the first file path is determined in the preset file path set.
[0045] In a possible design, the current time is the current date, and the preset file path set includes a first path and a second path;
[0046] The creation module is specifically used for:
[0047] Determine the difference between the current date and the preset date as the date difference;
[0048] If the date difference is an odd number, determining the first path in the preset file path set as the first file path;
[0049] If the date difference is an even number, the second path in the preset file path set is determined as the first file path.
[0050] In a possible design, the update module is specifically used for:
[0051] Determining a preset data volume of the new full amount of data;
[0052] If the data amount in the second data file is equal to the preset data amount, the file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
[0053] In one possible design, the update module is further used for:
[0054] Deleting historical data in the first data file;
[0055] Delete the first data file.
[0056] In one possible design, the historical data in the first data file is stored in at least one node;
[0057] The update module is specifically used for:
[0058] Each of the nodes is traversed in sequence, and the historical data in each of the nodes is deleted in sequence.
[0059] In a possible design, each of the nodes includes at least one partition, and the historical data in the first data file is stored in each of the partitions of each of the nodes;
[0060] The update module is specifically used for:
[0061] For any of the nodes, the partitions in the node are traversed in sequence, and the historical data in the partitions are deleted in sequence.
[0062] In a third aspect, an embodiment of the present application provides a data updating device, including:
[0063] Memory, used to store programs;
[0064] A processor is used to execute the program stored in the memory. When the program is executed, the processor is used to execute the method described in the first aspect and any one of the various possible designs of the first aspect.
[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any one of the various possible designs of the first aspect.
[0066] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method described in the first aspect and any one of the various possible designs of the first aspect.
[0067] The embodiment of the present application provides a data updating method and device, which includes: obtaining new full data of a first business. Determine a first data file corresponding to the first business, the first data file includes historical data of the first business, the file name of the first data file is a first name corresponding to the first business, and the first name is used to access the data corresponding to the first business. Create a second data file, and store the new full data in the second data file. Update the file name of the first data file to a name other than the first name, and update the file name of the second data file to the first name. By creating a second data file, the new full data is directly stored in the second data file, and after the storage is completed, the file name of the second data file is updated to the first name for data query. Then, before the file is renamed, the data can be queried from the first data file, and after the file is renamed, the data can be queried from the second data file, so that it can be ensured that during the entire process of data update, the data query processing can always be realized to effectively ensure the data stability during the data update process. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0069] Figure 1 A schematic diagram of a scenario of a data updating method provided in an embodiment of the present application;
[0070] Figure 2 A flowchart of a data updating method provided in an embodiment of the present application;
[0071] Figure 3The process of the data updating method provided in the embodiment of the present application Figure 2 ;
[0072] Figure 4 A schematic diagram of implementing the data file renaming provided in an embodiment of the present application;
[0073] Figure 5 A schematic diagram of determining that a path is occupied provided in an embodiment of the present application;
[0074] Figure 6 A schematic diagram of an implementation of determining a first file path provided in an embodiment of the present application;
[0075] Figure 7 A schematic diagram of an implementation of deleting historical data in the prior art provided in an embodiment of the present application;
[0076] Figure 8 Schematic diagram of polling and deleting historical data provided in the embodiment of the present application
[0077] Fig. 9 A flowchart of a data updating method provided in an embodiment of the present application;
[0078] Fig.10 A schematic diagram of the structure of a data updating device provided in an embodiment of the present application;
[0079] Fig.11 A schematic diagram of the hardware structure of the data updating device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0081] In order to better understand the technical solution of the present application, the relevant technologies involved in the present application are further introduced in detail below.
[0082] With the continuous development of Internet technology, a large amount of data will be stored in various platforms and systems. The data stored in them usually need to be updated. The data update operations can be divided into two types: full update and incremental update.
[0083] Incremental update means that when performing an update operation, only the data that needs to be changed is updated, and data that does not need to be updated or has already been updated will not be updated repeatedly. The full update method means that all historical data stored in the system needs to be updated.
[0084] In one possible implementation, the data in the system or platform can be stored in ClickHouse, for example, where ClickHouse is a columnar database management system for real-time analysis of big data, which shows very high efficiency in ad hoc queries.
[0085] By using vectorized execution and the underlying CPU instruction set (SIMD), ClickHouse can process massive amounts of data in parallel, thereby speeding up data processing. It is a common engine choice for online analytical processing (OLAP) applications. SIMD, or Single Instruction, Multiple Data, means that one instruction operates on multiple data.
[0086] Currently, there are two ways to synchronize data to Clickhouse, namely incremental processing and full processing. Incremental processing is relatively simple, and incremental data can be pushed every day. In the process of full processing, pushing full data involves deleting historical data, especially when the amount of historical data is large, the deletion operation is more troublesome.
[0087] The following is an example of incremental update and full update. For example, data can be stored in corresponding data tables. For example, the user behavior wide table in the e-commerce platform is an incremental table used to analyze the consumption behavior of logged-in users every day, which can be processed by incremental update.
[0088] For example, the wide table for new and old users in the e-commerce platform is a full table. The wide table for new and old users can store user information, order status, etc., because user information and order status may change. For example, user A placed an order to purchase product 1 yesterday, but applied for a return today, the order status will change accordingly, and this change is irregular and unpredictable. Therefore, in order to ensure the accuracy and effectiveness of the stored data, the wide table for new and old users needs to be fully updated.
[0089] The wide table of new and old users can be partitioned by month, for example. For example, multiple partitions can be set up, one partition corresponds to the data of one month, and then the full partition is dynamically updated every day. The wide table of new and old users can be used to analyze the order behavior of all new and old users.
[0090] However, the amount of data in the new and old user tables is very large every day, and the full update requires deleting the original historical data before data import.
[0091] In the current related technology, when implementing a full update, in one possible implementation method, the original data is deleted before the data is imported.
[0092] However, in this implementation, the original data is deleted and then synchronized. During the period of data synchronization, the front-end business cannot query the data. The user needs to wait until the data synchronization is completed before the query is successful. In particular, data synchronization occurs during the peak period of data query. If the synchronization time is long, the data query request also needs to wait for a long time, resulting in a lack of data availability and stability.
[0093] In another possible implementation, a Time To Live (TTL) statement may be used to set the life cycle of a table when creating the table.
[0094] This implementation method can be considered if the table data is relatively small, but if the data in the table is large, for example, more than 100G, due to the parameter restrictions of the platform, errors will often be reported, and the partition size cannot be deleted after exceeding 100G. And for the wide tables of new and old users introduced above, since they use dynamic partitions, this method is not applicable to this situation, so this implementation method will also lead to the lack of stability in data processing.
[0095] In response to the problems in the prior art, the following technical concept is proposed in this application: by constructing a temporary table to store data, when updating the full data, the full data is directly stored in the temporary table. After the data storage is completed, the table names of the temporary table and the formal table are replaced, and then the historical data is deleted, thereby ensuring that during the data update process, the data query request can always query the corresponding data, thereby effectively improving the data stability during the data update process.
[0096] The data updating method provided by the present application is introduced below in conjunction with specific embodiments. Figure 1 The scenario of the data updating method provided by this application is described. Figure 1 A schematic diagram of a scenario of a data updating method provided in an embodiment of the present application.
[0097] like Figure 1 As shown, the data in this application can be stored in Figure 1 In the data storage unit shown in , the data storage unit includes multiple nodes, for example Figure 1Among the nodes 1, 2, 3, ..., m, etc. shown in FIG. , one node may be understood as a device, such as a storage device, a server, etc.
[0098] And, each node can include multiple partitions, for example, node 1 includes Figure 1 Shown are Partition 1, Partition 2, ..., Partition n, and so on.
[0099] In a possible implementation, data may be specifically stored in each partition, for example, when the data to be stored is the wide table of new and old users introduced above, data for one month may be stored in one partition.
[0100] In the actual implementation process, the number of nodes and partitions included in the data storage unit, as well as the specific data content stored in each partition, can be selected according to actual needs, and this embodiment does not impose any restrictions on this.
[0101] The data updating method involved in the present application is described in detail below in conjunction with specific embodiments. It is worth noting that the execution subject of each embodiment in the present application can be, for example, a server, a processor, a microprocessor and other devices with data processing functions. This embodiment does not limit the specific implementation of the execution subject, which can be selected according to actual needs, as long as it is a device with data processing functions.
[0102] First combine Figure 2 Make an introduction, Figure 2 A flowchart of a data updating method provided in an embodiment of the present application.
[0103] like Figure 2 As shown, the method includes:
[0104] S201. Obtain new full data of the first business.
[0105] In this embodiment, when updating data, the data of the first business is updated. For example, for an e-commerce platform, the first business may be an order business, or an address information business, etc. In this embodiment, there is no restriction on the specific implementation method of the first business. It can be understood that a platform or a system is usually composed of many businesses, and each business corresponds to its own data. In this embodiment, there is no special restriction on the specific implementation method of the first business.
[0106] Currently, the data of the first service needs to be updated, and the data update in this embodiment is a full update, so new full data of the first service can be obtained.
[0107] For example, for the order business, the new full data of the first business may include all order business data before the current moment, such as order status, order time, order content, etc., and the new full data must be the latest data to ensure the accuracy of the data after the full update.
[0108] This embodiment does not limit the specific content and detailed implementation of the new full data, which depends on the implementation of the current first business.
[0109] S202: Determine a first data file corresponding to a first business, where the first data file includes historical data of the first business, and the file name of the first data file is a first name corresponding to the first business, and the first name is used to access data corresponding to the first business.
[0110] In this embodiment, a first data file corresponding to the first business can be determined. The first data file includes historical data of the first business. It can be understood that the historical data here refers to the new full data introduced above.
[0111] For example, there is a system that starts recording data on January 1st, and the data is updated on May 18th. The new full data may include the full data between January 1st and May 18th, and the historical data may include the full data from January 1st to May 17th.
[0112] The historical data stored in the first data file may have some changes in the data content compared to the new full data currently introduced, so a full update is required.
[0113] In this embodiment, the file name of the first data file is the first name, and its first name corresponds to the first business. For example, the data corresponding to the first business can be accessed through the first name. For example, the file name of the first data file is A. When the user requests to access the data of the first business, the user can query the data of the first business by accessing the file corresponding to the file name A.
[0114] This embodiment does not limit the specific implementation of the first name, as long as it can indicate access to the data of the first service.
[0115] S203: Create a second data file, and store the new full data in the second data file.
[0116] In order to avoid the need in the related art to synchronize the full amount of data only after the historical data is deleted, a second data file may be created in this embodiment, wherein the second data file may be used to store the new full amount of data.
[0117] The second data file in this embodiment can be understood as a temporary data file, and the first data file introduced above can be understood as a formal data file.
[0118] After the second data file is created, it does not contain any data. In this embodiment, the new full data can be stored in the second file without waiting for the data in the first data file to be deleted.
[0119] S204: Update the file name of the first data file to a name other than the first name, and update the file name of the second data file to the first name.
[0120] After the new full data is stored in the second data file, the second file data includes the latest full data of the first business. Therefore, in this embodiment, the file name of the second data file can be replaced and the second data file can be used as a file for data access.
[0121] Specifically, because the data for the first business is accessed based on the first name, the file name of the second data file can be updated to the first name. Assuming that the original file name of the second data file is B, the file name of the second data file can be replaced with A in this embodiment. The renamed second data file will be used as the current official file for querying the full amount of data and querying the full amount of data.
[0122] It is also understandable that two files with the same name usually cannot appear in one system, so in this embodiment, the file name of the first data file also needs to be updated to a name other than the first name.
[0123] For example, before the files are renamed, the file name of the first data file is A, and the file name of the second data file is B. Then, for example, the file name of the first data file may be first replaced with C, and then the file name of the second data file may be replaced with A.
[0124] It is worth mentioning that the processing speed of file renaming is very fast. Therefore, for the previous data query request, data query is performed in the first data file before the file is renamed, and data query is performed in the second data file after the file is renamed. In other words, during the data update process, the data query operation will not be affected in any way, and there is no need to wait for the historical data to be deleted. Data query can only be performed after the full data update is completed, which can effectively improve the data stability during the data update process.
[0125] The data updating method provided in the embodiment of the present application includes: obtaining new full data of the first business. Determine a first data file corresponding to the first business, the first data file includes historical data of the first business, the file name of the first data file is a first name corresponding to the first business, and the first name is used to access the data corresponding to the first business. Create a second data file, and store the new full data in the second data file. Update the file name of the first data file to a name other than the first name, and update the file name of the second data file to the first name. By creating a second data file, the new full data is directly stored in the second data file, and after the storage is completed, the file name of the second data file is updated to the first name for data query. Then, before the file is renamed, the data can be queried from the first data file, and after the file is renamed, the data can be queried from the second data file, thereby ensuring that data query processing can always be implemented throughout the entire process of data update, so as to effectively ensure data stability during the data update process.
[0126] Based on the above embodiments, the following Figures 3 to 6 The data update method is further introduced in detail. Figure 3 The process of the data updating method provided in the embodiment of the present application Figure 2 , Figure 4 A schematic diagram of implementing the data file renaming provided in the embodiment of the present application, Figure 5 A schematic diagram of determining that a path is occupied provided in an embodiment of the present application, Figure 6 A schematic diagram of an implementation of determining a first file path provided in an embodiment of the present application.
[0127] like Figure 3 As shown, the method includes
[0128] S301. Obtain new full data of the first service.
[0129] The implementation of S301 is similar to that of S201 and will not be described in detail here.
[0130] S302: Determine a first data file corresponding to a first business, where the first data file includes historical data of the first business, and the file name of the first data file is a first name corresponding to the first business, and the first name is used to access data corresponding to the first business.
[0131] The implementation of S302 is similar to that of S202. Here, the first data file and the first name are further described in detail with reference to a specific example.
[0132] The first data file in this embodiment can be understood as a formal file. For example, the file name of the first data file in this embodiment can be, for example, the physical formal table da_local, and the corresponding first name is the physical formal table da_local. When there is a data access request for the first business, the file is accessed specifically according to the name of "physical formal table da_local", thereby realizing data access to the first business.
[0133] In the actual implementation process, the specific setting of the first name can be selected according to actual needs, as long as the first name can indicate access to the data corresponding to the first service.
[0134] S303: Obtain the current time and a preset file path set, where the preset file path set includes a first file path, and the first file path is different from the file path of the first data file.
[0135] Furthermore, based on the above introduction, it can be determined that in this embodiment, a second data file needs to be created. In a possible implementation, when creating a file, in order to ensure the orderliness and manageability of the file, it is usually necessary to create the file under a certain file path.
[0136] In this embodiment, the second data file needs to be created in each time unit, and one time unit may be, for example, one day. For example, if the second data file is created today, it needs to be created tomorrow. In this embodiment, the second data file will be renamed. After the second data file is renamed, it will be stored in the system as a formal table.
[0137] This will result in, for example, a second data file is created under path 1 today, and then the second data file is saved in the system under path 1. If the second data file is created under path 1 again tomorrow, it will be found that path 1 is occupied. Therefore, the second data file cannot always be created under the same file path.
[0138] Therefore, a preset file path set is provided in this embodiment, and the preset file path set may include at least two file paths. Among the at least two file paths, at least the first file path is included, and the first file path is the path for currently creating the second data file.
[0139] And the first file path in this embodiment is different from the file path of the first data file, because this specifically involves the renaming operation of the second data file, so here firstly combines Figure 4 The name update of the second data file is introduced.
[0140] See also Figure 4, assuming that there is currently a first data file, where the file name of the first data file is A, and the first data file includes historical data of the first business, and that a second data file is currently being created, where the file name of the second data file is B, and after the second data file is created, it does not include data, which is equivalent to an empty file.
[0141] In this embodiment, there is new full data of the first business. Since the second data file is currently created, the new full data can be directly stored in the second data file B without waiting for the data in the first data file A to be deleted.
[0142] After all the new full data is stored in the second data file B, data synchronization of the new full data is achieved, and then the file can be renamed.
[0143] In one possible implementation, because the second data file needs to update its file name to the first name of the first data file, and because two names with the same name cannot exist at the same time, the file name of the first data file A can be first updated to C, and then the file name of the second data file B can be updated to A, thereby achieving the update of the name of the second data file to the first name.
[0144] When there is a request for data access, the second data file can be directly accessed according to the first name.
[0145] At the same time, it can be understood that the file name of the second data file is updated to the first name A, and the second data file stores the full data, then the second data file actually becomes the new first data file, and when the data is updated in the next time unit, the full data stored in the second data file A becomes historical data, and the second data file A becomes the new first data file A, and then the above data update operation is repeated.
[0146] The above describes the data update process. For the first data file and the second data file, the first data file has a corresponding file path, and the second data file also has a corresponding file path. Because the first data file and the second data file are two different files, their corresponding file paths must be different.
[0147] For example, you can combine Figure 5 To understand the process of determining the file path of the second data file, see Figure 5For example, in time unit 1, the file path of the first data file A is path 1, and the file path of the second data file B is path 2. Then after executing the data update operation described above, the file path of the first data file C is path 1, and the file path of the second data file A is path 2, and the second data file A is actually the new first data file.
[0148] Then in the next time unit, that is, Figure 5 In the time unit 2 shown in , when determining the file path for creating the second data file, if you still create it under path 2, you will find that path 2 is occupied, specifically, it is occupied by the new first data file (that is, the second data file of the previous time unit), and the second data file cannot be created under path 2.
[0149] In a possible implementation, since the first data file C will be deleted, the path 1 will be released, so see Figure 5 , for example, a second data file may be created under path 1.
[0150] Therefore, when determining the file path for creating the second data file in the next time unit, it is necessary to ensure that the file path of the second data file is different from the file path of the first data file.
[0151] The first file path in this embodiment is the file path for creating the second data file currently. Therefore, the file path for the first data file set in this embodiment is different from the file path for the second data file. The purpose is to ensure that the selected first file path is available.
[0152] Therefore, in this embodiment, a preset file path set can be obtained, and in a possible implementation, when selecting a path to create a second data file from the preset file path set, it can be determined based on the current time, for example, so the current time also needs to be obtained.
[0153] As mentioned above, the time unit can be one day. In the actual implementation process, the specific time unit can be selected and set according to actual needs. For example, the time unit can be one week, which means that the data of the first business is fully updated once a week, or it can be one month, one year, etc. This embodiment does not impose any special restrictions on the specific setting of the time unit.
[0154] S304: Determine a first file path in a preset file path set according to the current time.
[0155] After the current time and the preset file path set are acquired, the first file path may be determined in the preset file path set according to the current time.
[0156] In a possible implementation, the preset file path set may include a first path and a second path, where the first path here can be understood as path 1 introduced in the above example, and the second path here can be understood as path 2 introduced in the above example.
[0157] And the current time in this embodiment may be, for example, the current date, such as a certain month and a certain day.
[0158] In a possible implementation of determining the first file path from a preset file path set according to the current time, for example, the difference between the current date and the preset date can be determined as the date difference, and then the first file path is determined according to the date difference.
[0159] The preset date introduced here can be, for example, the starting date for storing the full data. For example, the first business starts to generate data on October 20, 2020, and the full data of the first business is stored starting from October 20, 2020, and then continuously updated. For example, October 20, 2020 can be used as the preset date.
[0160] Alternatively, any date may be used as the preset date in this embodiment, and this embodiment does not impose any limitation on this.
[0161] In a possible implementation manner of determining the first file according to the date difference, if the date difference is an odd number, a first path in the preset file path set is determined as the first file path.
[0162] In another possible implementation, if the date difference is an even number, the second path in the preset file path set is determined as the first file path.
[0163] For example, you can combine Figure 6 In the above implementation method, for example, if the preset date is 2002-10-20, if the current log is 2020-10-21, it can be determined that the date difference is 1. Assuming that the first path in the preset file path set is path 1, path 1 can be determined as the first file path.
[0164] For example, if the preset date is 2002-10-20, if the current log is 2020-10-22, it can be determined that the date difference is 2. Assuming that the second path in the preset file path set is path 2, path 2 can be determined as the first file path.
[0165] For another example, if the preset date is 2002-10-20, if the current log is 2020-10-23, it can be determined that the date difference is 3. Assuming that the first path in the preset file path set is path 1, path 1 can be determined as the first file path.
[0166] For example, if the preset date is 2002-10-20, if the current log is 2020-10-24, it can be determined that the date difference is 4. Assuming that the second path in the preset file path set is path 2, path 2 can be determined as the first file path.
[0167] By analogy, it can be understood that, based on the above-described implementation method of determining the first file path from the preset file path set according to the date difference, it can be ensured that the first file path selected today is different from the first file path selected yesterday. Figure 6 In the example, path 1 and path 2 are actually selected by polling.
[0168] However, it should be noted that the above-mentioned implementation method of selecting the first file path based on the parity of the date difference is only one possible implementation method. In other possible implementation methods, for example, a plurality of file paths may be included in the preset file path set, such as 3, 4, etc., and then polling and selecting among these multiple file paths is performed to ensure that the file path selected each day is different from that of the previous day.
[0169] In the actual implementation process, the implementation method of determining the first file path from the preset file path set can be selected according to actual needs, as long as the first file paths selected in adjacent time units are different, and this embodiment does not limit its specific implementation method.
[0170] S305: Create a second data file according to the first file path.
[0171] S306: Store the new full data into the second data file.
[0172] After the first file path is determined, a second data file may be created according to the first file path, and the new full data may be stored in the second data file.
[0173] This ensures that the new full data file can be stored directly without waiting for the original historical data to be deleted, so as to ensure that data query can be effectively implemented during the data update process.
[0174] S307: Determine the preset data volume of the new full data.
[0175] In a possible implementation of this embodiment, when updating the file names of the first data file and the second data file, it is necessary to ensure that all new full data has been stored in the second data file.
[0176] And the data amount of the new full data can be determined, so the preset data amount of the new full data can be determined first.
[0177] S308: If the data volume in the second data file is equal to the preset data volume, the file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
[0178] Afterwards, based on the preset data volume of the new full data, it can be determined whether the synchronization of the full data is completed.
[0179] In one possible implementation, if it is determined that the amount of data in the second data file is equal to the preset data amount, it can be determined that all new full data is stored in the second data file, and further it can be determined that the synchronization of the full data is completed, after which the file renaming operation can be performed. Specifically, the file name of the first data file can be updated to a name other than the first name, and the file name of the second data file can be updated to the first name. A more detailed renaming operation has been introduced in detail in step S303 of this embodiment, and will not be repeated here.
[0180] In another possible implementation, if it is determined that the amount of data in the second data file is less than the preset data, it can be determined that the new full data has not been synchronized yet, and you can continue to wait. If the amount of data in the second data file is still less than the preset data amount after the preset time, you can stop the current operation and check the upstream synchronization task to determine whether an abnormality occurs.
[0181] The data updating method provided in the embodiment of the present application needs to determine the first file path corresponding to the second data file when creating the second data file. When determining the first file path, it is selected by polling in the preset file path set, so as to ensure that the file path selected in each time unit is different from the file path selected in the previous time unit, and then effectively ensure that the selected file path will not be occupied, ensuring that the second data file can be created effectively and correctly, and then directly storing the new full data in the second data file, so as to ensure that in the process of data update, data query for the first business can also be effectively realized. And in this embodiment, when updating the name of the file, it is to ensure that the new full data is stored in the second data file, and then perform the file renaming operation, so as to effectively ensure the correctness and integrity of data synchronization.
[0182] Based on the above description, it can be determined that in the process of updating the full data, in addition to synchronizing the full data, the historical data also needs to be deleted. Therefore, based on the above embodiment, the following Figure 7 and Figure 8 The specific implementation method for deleting historical data is introduced.
[0183] Figure 7 This is a schematic diagram of deleting historical data in the prior art provided in an embodiment of the present application. Figure 8 A schematic diagram of implementing polling to delete historical data provided in an embodiment of the present application.
[0184] See also Figure 7 and Figure 8 In this embodiment, the data storage structure may be, for example, a data storage unit including multiple nodes, a node may be understood as a storage device, each node includes multiple partitions, and the data is distributedly stored in each partition of each node.
[0185] In the prior art, when deleting data, the entire node is usually deleted directly. For example, see Figure 7 Currently, the data of each partition in these m nodes is deleted concurrently, so it is necessary to access these m nodes at the same time. However, due to the large amount of data and the large number of corresponding partitions, it is easy for the cluster to crash and report errors if all nodes are deleted directly, resulting in failure in deleting historical data.
[0186] In this embodiment, a node polling + partition polling method is adopted to first log in to each node, and then traverse and delete the data of each partition in the table of each node, and delete them slowly in small batches.
[0187] Specifically, each node may be traversed in sequence, and the historical data in each node may be deleted in sequence.
[0188] When deleting the historical data in each node, specifically, each partition in the node is traversed in sequence, and the historical data in each partition is deleted in sequence.
[0189] For example, you can combine Figure 8 To understand, Figure 8 It also includes multiple nodes, and each node includes multiple partitions. For example, partition 1 in node 1 can be deleted first (the first deletion operation), and then partition 2 in node 1 can be deleted (the second deletion operation), and so on, partition n in node 1 can be deleted (the nth deletion operation). At this point, the data in node 1 is deleted.
[0190] After the deletion of node 1 is completed, the data of each partition in node 2 is deleted. The specific deletion process is similar to the above-mentioned node 1, and the implementation method of each subsequent node is also similar.
[0191] The above deletion operation is a small batch sequential polling deletion. During the deletion operation, a polling detection mechanism can also be set to retry the nodes that failed to delete, thereby effectively solving the problem of failed data deletion.
[0192] At the same time, it can be understood that because the new full data in this embodiment is directly stored in the second data file, there is no need to wait for the historical data in the first data file to be deleted. Therefore, the speed requirement for deleting the historical data in the first data file is not very high, and it can be deleted slowly according to the process described above.
[0193] After the historical data in the first data file is deleted according to the above process, the first data file can also be deleted, and the file path corresponding to the corresponding first data file is released. The renamed second data file is then used as the new first data file, and the operations of the above embodiments are repeated in the next time unit.
[0194] The data updating method provided in this embodiment is provided with a second data file to store the new full data, so the historical data in the first data file can be deleted slowly. Specifically, in this embodiment, when deleting the historical data in the first data file, node polling and partition polling are adopted to slowly delete the historical data in the first data file in sequence, thereby avoiding the problem of cluster crashes and errors caused by direct deletion due to excessive data volume, and effectively ensuring the orderliness and correctness of the deleted data.
[0195] Based on the above embodiments, a specific example is given below. Fig. 9 A systematic introduction to the data updating method provided by this application is given. Fig. 9 A flowchart of a data updating method provided in an embodiment of the present application.
[0196] Fig. 9 The introduction in this section is based on the Clickhouse introduced above, and assumes that Fig. 9 In the example, a time unit is one day, which means that the full data is updated every day.
[0197] See also Fig. 9 , you can first perform the table initialization operation of the full table, for example, you can create the following table in Clickhouse:
[0198] Physical distributed table da, physical formal table da_local and temporary physical table tmp, among which the physical distributed table da is used to receive data access requests, that is, all data access requests are accessed to the physical distributed table da, and then the physical distributed table da is linked to the physical formal table da_local to realize the processing of data access requests.
[0199] In addition, the purpose of creating a new physical temporary table tmp is to prepare for renaming the table name to the physical formal table da_local.
[0200] During initialization, the file path of the newly created formal table may be, for example, path 1, and the file path of the newly created temporary table may be, for example, path 2.
[0201] Afterwards, when synchronizing the new full data, it is synchronized directly to the physical temporary table tmp, instead of writing to the physical formal table da_local every time.
[0202] Later, for example, you can encapsulate the update logic of the full data into a script, create a new computing task separately, and rely on the data synchronization task.
[0203] Specifically, you can first check whether the upstream synchronization task has successfully written the physical temporary table tmp of Clickhouse. If the data volume is normal, proceed to the next step. If the data volume is abnormal, exit and prompt to check the upstream synchronization task.
[0204] Then perform the table name replacement rename operation, which can include the following two parts:
[0205] The first part is to rename the physical formal table da_local to the backup table da_local_bak. However, it should be noted that although the table name has changed, the zookeeper file path dependency remains unchanged, which is still path 1. ZooKeeper is a distributed, open source distributed application coordination service. The metadata information of Clickhouse table creation is stored in the zookeeper node.
[0206] The second part is to replace the physical temporary table tmp with the physical formal table da_local. It should be noted that it is the file path or path 2.
[0207] Through these two operations, the temporary table and the formal table are replaced, and the data exchange in space is completed. The entire table name replacement process is completed within seconds and does not affect business queries at all. However, when renaming every day, the Zookeeper path will not change, only the table name will be modified, which will cause the Zookeeper path to be occupied when the table is created every day. The solution is described below.
[0208] The next step is to delete the backup table da_local_bak. However, since the amount of data in the table is very large, usually exceeding 100G, it is easy to get stuck in parallel when deleting partition data using the node routing on cluster method.
[0209] The optimization strategy adopted in this application is to use node polling + table partition polling on Clickhouse. A mechanism for deleting data in small batches is implemented, and a polling detection mechanism is set up to retry failed nodes, effectively solving the problem of failed data partition deletion. The specific approach is as follows:
[0210] First, get the latest ClickHouse node information through system.cluster and append it to file a. Then, deduplicate the dt partition of the full table and get the dt partition and append it to file b. System.cluster is the system configuration table in ClickHouse.
[0211] Then, for example, after logging in to the Clickhouse node through a Hypertext Transport Protocol (HTTP) request, file a and file b are parsed to obtain the IP address and dt partition, and then a double loop is performed to delete the data partition operation, and a small batch deletion is performed on each partition in the entire table of each node. The purpose of this operation is to greatly reduce the machine load and avoid the original situation of data deletion being stuck. For a more specific implementation method, please refer to the introduction in the above embodiment, which will not be repeated here.
[0212] After deleting the backup table data, delete the bak backup table, and path 1 will also be deleted.
[0213] After the operation is completed on the same day, the physical temporary table tmp needs to be automatically created the next day, but path 2 is already occupied, so the file path name needs to be changed to path 1. After the physical temporary table tmp is renamed on the second day, the physical temporary table tmp needs to be re-created on the third day, and the path name is newly created as path 2. Therefore, table creation needs to be polled every day, with the same table name but different paths. The paths involved in the polling may include path 1 and path 2, for example.
[0214] In the present application, for example, the file path for creating the second data file can be determined based on the date difference between the current date and the preset date.
[0215] For example, when the date difference is an odd number, path 1 is selected, and when the date difference is an even number, path 2 is selected. This date polling method can effectively solve the problem of path occupation.
[0216] Then, after a new physical temporary tmp table is created every day, the task is automatically executed the next day, repeating the process of the first step to form a closed loop of the entire process.
[0217] In summary, the data update method provided by the embodiment of the present application completes data exchange by replacing the table names of the temporary table and the formal table, which shows that the execution time of the replacement can be completed within seconds. Compared with the original need to delete data and then synchronize data, which will cause the business to be unable to query data for a long time, it can effectively ensure the stability of data during the data update process, thereby greatly improving the high availability of data and user experience. In addition, the node polling + partition polling method is used to perform data deletion operations, avoiding the problem of frequent jamming caused by large-scale parallel deletion, greatly improving the efficiency of deleting data.
[0218] Fig.10 This is a schematic diagram of the structure of the data updating device provided in the embodiment of the present application. Fig.10 As shown, the device 100 includes: an acquisition module 1001 , a determination module 1002 , a creation module 1003 and an update module 1004 .
[0219] An acquisition module 1001 is used to acquire new full data of a first service;
[0220] A determination module 1002 is configured to determine a first data file corresponding to the first service, wherein the first data file includes historical data of the first service, and the file name of the first data file is a first name corresponding to the first service, and the first name is used to access data corresponding to the first service;
[0221] A creation module 1003 is used to create a second data file and store the new full data in the second data file;
[0222] The updating module 1004 is used to update the file name of the first data file to a name other than the first name, and to update the file name of the second data file to the first name.
[0223] In a possible design, the creation module 1003 is specifically used for:
[0224] Determining a first file path, the first file path being different from a file path of the first data file;
[0225] The second data file is created according to the first file path.
[0226] In a possible design, the creation module 1003 is specifically used for:
[0227] Obtain the current time and a preset file path set, wherein the preset file path set includes the first file path;
[0228] According to the current time, the first file path is determined in the preset file path set.
[0229] In a possible design, the current time is the current date, and the preset file path set includes a first path and a second path;
[0230] The creation module 1003 is specifically used for:
[0231] Determine the difference between the current date and the preset date as the date difference;
[0232] If the date difference is an odd number, determining the first path in the preset file path set as the first file path;
[0233] If the date difference is an even number, the second path in the preset file path set is determined as the first file path.
[0234] In a possible design, the updating module 1004 is specifically used for:
[0235] Determining a preset data volume of the new full amount of data;
[0236] If the data amount in the second data file is equal to the preset data amount, the file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
[0237] In one possible design, the updating module 1004 is further configured to:
[0238] Deleting historical data in the first data file;
[0239] Delete the first data file.
[0240] In one possible design, the historical data in the first data file is stored in at least one node;
[0241] The updating module 1004 is specifically used for:
[0242] Each of the nodes is traversed in sequence, and the historical data in each of the nodes is deleted in sequence.
[0243] In a possible design, each of the nodes includes at least one partition, and the historical data in the first data file is stored in each of the partitions of each of the nodes;
[0244] The updating module 1004 is specifically used for:
[0245] For any of the nodes, the partitions in the node are traversed in sequence, and the historical data in the partitions are deleted in sequence.
[0246] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.
[0247] Fig.11 A schematic diagram of the hardware structure of the data updating device provided in the embodiment of the present application is shown in FIG. Fig.11 As shown, the data updating device 110 of this embodiment includes: a processor 1101 and a memory 1102;
[0248] Memory 1102, used to store computer-executable instructions;
[0249] The processor 1101 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the data updating method in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0250] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .
[0251] When the memory 1102 is independently provided, the data updating device further includes a bus 1103 for connecting the memory 1102 and the processor 1101 .
[0252] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data updating method executed by the above data updating device is implemented.
[0253] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0254] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.
[0255] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.
[0256] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.
[0257] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0258] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0259] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A data updating method, comprising: Obtain new full data of the first business; Determine a first data file corresponding to the first service, the first data file including historical data of the first service, the file name of the first data file being a first name corresponding to the first service, the first name being used to access data corresponding to the first service; Acquire the current time and a preset file path set, wherein the preset file path set includes at least two file paths, and the at least two file paths include at least the first file path; According to the current time, determining the first file path in the preset file path set, creating a second data file according to the first file path, and storing the new full data in the second data file; The first file path is different from the file path of the first data file; The file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
2. The method according to claim 1, characterized in that The current time is the current date, and the preset file path set includes a first path and a second path; The determining the first file path in the preset file path set according to the current time includes: Determine the difference between the current date and the preset date as the date difference; If the date difference is an odd number, determining the first path in the preset file path set as the first file path; If the date difference is an even number, the second path in the preset file path set is determined as the first file path.
3. The method according to claim 1, characterized in that The updating of the file name of the first data file to a name other than the first name, and the updating of the file name of the second data file to the first name, comprises: Determining a preset data volume of the new full amount of data; If the data amount in the second data file is equal to the preset data amount, the file name of the first data file is updated to a name other than the first name, and the file name of the second data file is updated to the first name.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Deleting historical data in the first data file; Delete the first data file.
5. The method according to claim 4, characterized in that The historical data in the first data file is stored in at least one node; Deleting the historical data in the first data file includes: Each of the nodes is traversed in sequence, and the historical data in each of the nodes is deleted in sequence.
6. The method according to claim 5, characterized in that Each of the nodes includes at least one partition, and the historical data in the first data file is stored in each of the partitions of each of the nodes; The deleting the historical data in each of the nodes in sequence includes: For any of the nodes, the partitions in the node are traversed in sequence, and the historical data in the partitions are deleted in sequence.
7. A data updating device, characterized in that: include: An acquisition module, used to acquire new full data of the first service; A determination module, configured to determine a first data file corresponding to the first service, wherein the first data file includes historical data of the first service, and the file name of the first data file is a first name corresponding to the first service, and the first name is used to access data corresponding to the first service; a creation module, configured to obtain a current time and a preset file path set, wherein the preset file path set includes at least two file paths, and the at least two file paths include at least the first file path; determine the first file path in the preset file path set according to the current time, create the second data file according to the first file path, and store the new full data in the second data file; The first file path is different from the file path of the first data file; An updating module is used to update the file name of the first data file to a name other than the first name, and to update the file name of the second data file to the first name.
8. A data updating device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.