Data scrubbing method and apparatus

Abstract

Embodiments of the present specification provide a data backwash method and device. The method comprises: obtaining modified calculation code; wherein the modified calculation code corresponds to data aggregation of a first granularity; decomposing a second granularity from the first granularity; wherein the second granularity is a finer granularity relative to the first granularity; obtaining a data backwash task according to the modified calculation code; wherein the data backwash task comprises: aggregating multiple data of the second granularity to obtain aggregation of data corresponding to the first granularity; obtaining all data of the second granularity required by the data backwash task according to the data backwash task; and performing data aggregation using the obtained all data of the second granularity. Embodiments of the present specification can improve the processing efficiency of data backwash.

Description

Technical Field

[0001] This specification relates to network communication technology in one or more embodiments, and more particularly to a data refresh method and apparatus. Background Technology

[0002] Data refresh refers to the process of supplementing or updating certain or all data over a period of time during data processing due to various reasons, such as adjustments to the statistical criteria for current transactions. For example, in a given year, the total number of times an application M was accessed in the past 30 days relative to that day was calculated for each day. However, due to changes in business requirements, such as the original statistics not counting visits to mini-program A, which is integrated within application M, but needing to achieve year-on-year or month-on-month comparisons to include visits to application M, it is necessary to recalculate the total number of times application M was accessed in the past 30 days relative to that day for each day of the year—this is data refresh.

[0003] Data refresh requires a significant amount of time to reread and recalculate large amounts of data, thus consuming excessive time and computing resources and reducing the efficiency of data refresh processing. Summary of the Invention

[0004] This specification describes a data refresh method and apparatus through one or more embodiments, which can improve the processing efficiency of data refresh.

[0005] According to the first aspect, a data refresh method is provided, wherein the method includes:

[0006] The modified calculation code is obtained; where the modified calculation code corresponds to the summary of the data at the first granularity.

[0007] A second particle size is derived from the first particle size; wherein the second particle size is a finer particle size relative to the first particle size.

[0008] Based on the modified calculation code, a data refresh task is obtained; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the data corresponding to the first granularity.

[0009] Based on the data refresh task, obtain all the second-granularity data required by the data refresh task;

[0010] In each time period, multiple data points of the second granularity required for the current time period are selected from all the acquired second-granularity data and the data is summarized.

[0011] The step of obtaining all the second-granularity data required for the data rollback task includes: reading all the second-granularity data required for the data rollback task from the data warehouse at once.

[0012] Prior to reading all the second-granularity data required for the data flushing task from the data warehouse in one go, the process further includes:

[0013] Determine whether the data warehouse contains all the second-granularity data required for the data rollback task;

[0014] If so, continue with the step of reading all the second-granularity data required for the data flushing task from the data warehouse in one go;

[0015] If not, obtain the task code for retrieving the second granularity data input by the administrator; run the task code for retrieving the second granularity data to obtain all the second granularity data required by the data refresh task.

[0016] The aggregation of the first granularity of data includes: summarizing the data from the second preset time period before the current time period in each time cycle within the first preset time period.

[0017] The step of selecting multiple data points of the second granularity required for the current time period from all acquired second-granularity data in each time period and summarizing the data includes:

[0018] In each time period within the first preset time period, multiple data points of the second granularity required in the current time period are extracted from all the read second granularity data using a sliding window method;

[0019] Using the extracted data of the second granularity required within the current time period, the result of summing up the data of the second granularity within the second preset time period before the current time period is calculated.

[0020] The result of summing up multiple data points of the second granularity within the second preset time period before the current time period is taken as the data summary result of the first granularity within the second preset time period before the current time period.

[0021] The result of accumulating multiple data points of the second granularity within the second preset time period prior to the current time period includes:

[0022] From the summary result of the previous time period, i.e., the result of accumulating multiple data of the second granularity within the second preset time period before the previous time period, subtract the second granularity data of the first date required for the previous time period, and add the second granularity data of the last date required for the current time period.

[0023] The first granularity of data includes: the total number of visits to the specified platform across the entire network; the second granularity of data includes: the number of visits to all merchants or all applications on the specified platform.

[0024] or,

[0025] The data at the first granularity includes: the total number of visits to the specified platform within a third preset time period; correspondingly, the data at the second granularity includes: the total number of visits to the specified platform within each fourth preset time period; wherein, the third preset time period is equal to the sum of multiple fourth preset time periods.

[0026] According to the second aspect, a data refresh device is provided, wherein the device includes:

[0027] Modify the code acquisition module to obtain the modified calculation code; where the modified calculation code corresponds to the summary of the first granularity of data.

[0028] The granularity decomposition module is configured to decompose a second granularity from a first granularity; wherein the second granularity is a finer granularity than the first granularity.

[0029] The data refresh task acquisition module is configured to obtain a data refresh task based on the modified calculation code; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the corresponding first granularity data;

[0030] The fine-grained data acquisition module is configured to acquire all second-granularity data required by the data refresh task based on the data refresh task.

[0031] The aggregation module is configured to select multiple data points of the second granularity required for the current time period from all acquired second-granularity data in each time period for data aggregation.

[0032] The aggregation of the first granularity data includes: summarizing the aggregated data of the first granularity in each time period before the current time period in each time cycle of the first preset time period.

[0033] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0034] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in any embodiment of this specification.

[0035] The data flushing method and apparatus for data aggregation services provided in the various embodiments of this specification modify the implementation logic of data flushing services in the prior art. Instead of repeatedly executing the modified calculation code for each time period, it eliminates the need for the prior art to perform the data acquisition process required for aggregation once in each time period. This is because the prior art's processing leads to a large amount of repetitive data reading and calculation. In the embodiments of this specification, after obtaining the data flushing task based on the modified calculation code, all the second-granularity data required for the data flushing task are acquired at once. Subsequently, in each time period, the data required for the current time period is directly selected from all the acquired second-granularity data for data aggregation. Therefore, the embodiments of this specification change the aggregation of coarse-grained data to the aggregation of multiple fine-grained data, thereby facilitating data retrieval from the data warehouse and subsequent aggregation calculations in periodic data aggregation services. It is evident that the embodiments of this specification avoid a large amount of repetitive data reading and calculation, improving the efficiency of data flushing services. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a data refresh method for data aggregation-type businesses in existing technologies.

[0038] Figure 2 This is a schematic diagram of the system architecture used in one embodiment of this specification.

[0039] Figure 3 This is a flowchart of a data refresh method for data aggregation services in one embodiment of this specification.

[0040] Figure 4 This is a schematic diagram of the data refresh device for data aggregation services in one embodiment of this specification. Detailed Implementation

[0041] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0042] First, it should be noted that the terminology used in the embodiments of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0043] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0044] In the prior art, see Figure 1 Methods for data rollback in data aggregation-related businesses include:

[0045] Step 101: Modify the calculation code used for data aggregation in response to changes in business requirements.

[0046] Step 102: When summarizing the data, the modified calculation code is executed repeatedly for each time period in order to read and calculate the new data summary results.

[0047] For example, a data refresh scenario might involve calculating the total number of times an application M was accessed in the past 30 days for each time period in 2022, such as each day. Due to changes in business requirements, such as the previous calculation not counting accesses to mini-program A within application M as accesses to application M, but now needing to include accesses to mini-program A for year-on-year or month-on-month comparisons, it's necessary to recalculate the total number of times application M was accessed in the past 30 days for each day in 2022 – this is known as data refresh.

[0048] according to Figure 1The existing method described first modifies the calculation code used for data statistics, for example, changing the data filtering condition from excluding data from mini-program A to including data from mini-program A. Then, the modified calculation code is executed in the calculation node. That is, for January 1, 2022, the modified calculation code is re-executed to read the total number of times application M was accessed in the past 30 days relative to January 1, 2022 from the data warehouse. Next, for January 2, 2022, the modified calculation code is re-executed to read the total number of times application M was accessed in the past 30 days relative to January 2, 2022 from the data warehouse. This process continues until December 31, 2022, when the modified calculation code is re-executed to read the total number of times application M was accessed in the past 30 days relative to December 31, 2022 from the data warehouse.

[0049] Existing methods lead to significant waste of data reading resources because data acquired across different time periods often overlaps. Current techniques do not account for this overlap, acquiring and processing data independently for each time period. For example, for January 1st and 2nd, 2022, data needs to be read from the data warehouse twice, but much of the data read is identical, resulting in a large amount of duplicate data reading. For January 1, 2022, the data read during the calculation was from the 30 days from December 2 to December 31, 2021. For January 2, 2022, the data read during the calculation was from the 30 days from December 3, 2021 to January 1, 2022. It can be seen that in both data reads, the data from December 3 to December 31, 2021 was read. In other words, for each day's calculation, the same 29 days of data were read compared to the previous day, resulting in a significant waste of reading time.

[0050] Furthermore, existing methods can lead to a waste of computing resources. For example, in both data reads, data from December 3, 2021 to December 31, 2021 (29 days) was read. This means that for each day's calculation, the data for 29 days was repeated relative to the previous day, resulting in a significant waste of computing resources.

[0051] To facilitate understanding of the methods provided in this manual, the system architecture involved and applicable to this manual is first described. For example... Figure 2 As shown, the system architecture mainly includes three types of network nodes: web crawler module, data warehouse, and data refresh device.

[0052] The web crawler module is used to crawl various types of data from the web and store the crawled data in a data warehouse.

[0053] A data warehouse, also known as a data platform, is a platform for offline data processing. Within a data warehouse / data platform, data is cleaned, processed, and integrated.

[0054] A data refresh device is used to refresh data using data from a data warehouse.

[0055] It should be understood that Figure 2 The number of web crawler modules, data warehouses, and data refresh devices shown is merely illustrative. Any number can be selected and deployed as needed.

[0056] Figure 3 This is a flowchart of a data refresh method for data aggregation services, as described in one embodiment of this specification. The execution entity of this method is a data refresh device for data aggregation services. It is understood that this method can also be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. See also... Figure 3 The method includes:

[0057] Step 301: Obtain the modified calculation code; where the modified calculation code corresponds to the summary of the data at the first granularity.

[0058] Step 303: Decompose the first particle size into a second particle size; wherein the second particle size is a finer particle size relative to the first particle size.

[0059] Step 305: Based on the modified calculation code, a data refresh task is obtained; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the data corresponding to the first granularity.

[0060] Step 307: Based on the data refresh task, obtain all the second-granularity data required by the data refresh task.

[0061] Step 309: In each time period, select multiple data points of the second granularity required for the current time period from all the acquired second granularity data and summarize the data.

[0062] According to the above Figure 3As can be seen from the illustrated process, the data rollback method of this embodiment modifies the implementation logic of data rollback operations in the prior art. Instead of repeatedly executing the modified calculation code for each time period, it avoids the need for the prior art to perform the data acquisition and summarization process once in each time period (data acquired in different time periods often overlaps). This is because such processing in the prior art leads to a large amount of repetitive data reading and calculation. In this embodiment, after obtaining the data rollback task based on the modified calculation code, all the second-granularity data required for the data rollback task are acquired at once. Subsequently, in each time period, the data of the second-granularity required for the current time period is directly selected from all the acquired second-granularity data for data summarization. Therefore, this embodiment changes the summarization of coarse-grained data to the summarization of multiple fine-grained data, thereby facilitating data reading from the data warehouse and subsequent calculations in periodic data summarization operations. The method of this embodiment avoids a large amount of repetitive data reading and calculation, improving the efficiency of data rollback operations.

[0063] The following examples illustrate this point. Figure 3 The methods shown in the embodiments of this specification will be described.

[0064] First, for step 301: obtain the modified calculation code; where the modified calculation code corresponds to the summary of the data at the first granularity.

[0065] If, for some reason, such as an adjustment to the statistical caliber of current transactions, data needs to be supplemented or updated over a period of time, data backflush is required, and administrators will modify the corresponding calculation code. In step 301, the modified calculation code is obtained. This modified calculation code corresponds to the summary of the first granularity of data.

[0066] Next, for step 303: decompose the first granularity into a second granularity; wherein the second granularity is a finer granularity relative to the first granularity.

[0067] In a data warehouse, data is granular. To better utilize the existing data in the data warehouse, this embodiment changes the aggregation of coarse-grained data to the aggregation of the summation results of multiple fine-grained data, thereby facilitating data reading and calculation during periodic data aggregation. In this embodiment, the first granularity is relatively coarse-grained, and the second granularity is relatively fine-grained. For example, coarse-grained data (the first granularity) includes: the total number of visits to the entire network on a specified platform; fine-grained data (the second granularity) includes: the number of visits to all merchants or all applications on the specified platform.

[0068] For example, the first granularity of data includes: the total number of visits to the specified platform across the entire network within a third preset time period; correspondingly, the second granularity of data includes: the total number of visits to the specified platform across the entire network within each fourth preset time period; wherein, the third preset time period is equal to the sum of multiple fourth preset time periods. For instance, the first granularity of data includes: the total number of times application M was accessed within the past 30 days relative to each day in each time period of 2022; the finer granularity, i.e., the second granularity of data, includes: the total number of times application M was accessed on each day within the past 30 days relative to each day in each time period of 2022. In other words, the 30 days in the first granularity are broken down into the sum of each day within the 30 days of the second granularity.

[0069] Next, for step 305: Based on the modified calculation code, a data refresh task is obtained; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the data corresponding to the first granularity.

[0070] As mentioned earlier, in the data refresh for periodic data aggregation, the embodiments of this specification change the aggregation of coarse-grained data to the aggregation of the cumulative results of multiple fine-grained data.

[0071] Since the second granularity is derived from the first granularity, the data flushing task obtained in step 305 is equivalent to changing the summary flushing task for the original coarse-grained data to a new summary flushing task for the fine-grained data, which is either a summation or a counting task. In other words, the summary of the first granularity data can be obtained by summing or counting multiple second-granularity data.

[0072] Next, for step 307: Based on the data refresh task, obtain all the second-granularity data required by the data refresh task.

[0073] In this embodiment, the required data is not acquired separately for each time period, thus avoiding duplicate data reading caused by independent data reading in each time period. In step 307, all the second-granularity data required for the data rollback task can be read from the data warehouse at once. For example, the data rollback task includes: for each day in 2022, summing the total number of times application M was accessed in the past 30 days relative to that day, and then aggregating the results. Therefore, all the second-granularity data required for the data rollback task is the total number of times application M was accessed in each day from December 2, 2021 to December 30, 2022. Then, all the required second-granularity data can be read from the data warehouse at once.

[0074] Data warehouses store various types of data, typically including fine-grained data. For all the second-granularity data required for a data rollback task, in some cases, the data warehouse already contains this data, allowing for direct retrieval; however, in other cases, the data warehouse does not contain all of this data. Therefore, in step 307, before executing the one-time reading of all the second-granularity data required for the data rollback task from the data warehouse, the following further steps are included:

[0075] Step 3070: Determine whether the data warehouse contains all the second-granularity data required for the data rollback task. If yes, proceed to step 3071; otherwise, proceed to step 3073.

[0076] Step 3071: Read all the second-granularity data required for the data flushing task from the data warehouse at once, and end the process of step 307.

[0077] Step 3073: Obtain the task code for retrieving the second granularity data input by the administrator.

[0078] Step 3075: Run the task code to obtain the second granularity data to get all the second granularity data required by the data flushing task.

[0079] For example, if the data warehouse lacks all the second-level data required for a data rollback task—the total number of times application M was accessed each day from December 2, 2021 to December 30, 2022—then the administrator can be prompted to enter the corresponding task code. Executing this task code will retrieve the total number of times application M was accessed each day from December 2, 2021 to December 30, 2022.

[0080] Next, for step 309: In each time period, select multiple data points of the second granularity required for the current time period from all the acquired second-granularity data and summarize the data.

[0081] In one embodiment of this specification, data refresh is applicable to periodic data aggregation services. That is, the aggregation of data at the first granularity includes: summarizing the aggregation results of data within a second preset time period prior to the current time period in each time cycle within a first preset time period. For example, for each day in 2022, summarizing the total number of times application M was accessed within the past 30 days relative to that day. Accordingly, the resulting data refresh task is a cumulative aggregation of data at multiple second granularities, including: for each day in 2022, summarizing the total number of times application M was accessed within each of the past 30 days relative to that day (that is, the 30 days in the first granularity equals the sum of each day within the 30 days in the second granularity).

[0082] In step 309, within each time period, multiple data points of the required second granularity for the current time period are selected from all acquired second-granularity data and aggregated, including:

[0083] Step 3091: In each time period within the first preset time period, use a sliding window to extract multiple data points of the second granularity required for the current time period from all the read second granularity data.

[0084] In step 3091, by employing a sliding window approach, recurring data required across different time periods can be retrieved without repeatedly searching and reading from the data warehouse. Instead, the sliding window can be moved progressively from all the second-granularity data acquired at once. For example, all second-granularity data includes the total number of times application M was accessed each day from December 2, 2021 to December 31, 2022, totaling 365 + 30 = 395 days of data. For the first day of 2022, January 1, 2022, the sliding window starts from December 2, 2021, and lasts for 30 days. Data for those 30 days is retrieved, specifically the total number of times application M was accessed from December 2, 2021, to December 31, 2021. For the second day of 2022, January 2, 2022, the sliding window moves forward one day, starting from December 3, 2021, and lasts for 30 days. Data for those 30 days is retrieved, specifically the total number of times application M was accessed from December 3, 2021, to January 1, 2022, and so on.

[0085] Step 3093: Using the extracted data of the second granularity required within the current time period, calculate the result of the sum of the data of the second granularity within the second preset time period before the current time period.

[0086] For example, for the first day of 2022, January 1, 2022, the total number of times application M was accessed from December 2, 2021 to December 31, 2021 is accumulated; for the second day of 2022, January 2, 2022, the total number of times application M was accessed from December 3, 2021 to January 1, 2022 is accumulated, and so on, to obtain the required summary result.

[0087] In one embodiment of this specification, to further reduce the computational load, the second-granularity data of the first date required for the previous time period can be subtracted from the summary result of the previous time period (which is the result of accumulating multiple data of the second granularity within the second preset time period before the previous time period), and the second-granularity data of the last date required for the current time period can be added. For example, it can be seen that, relative to January 1, 2022, the statistical result required for January 2, 2022 is equal to the statistical result of January 1, 2022 minus the total number of times application M was accessed on December 2, 2021, plus the total number of times application M was accessed on January 1, 2022.

[0088] Step 3095: The result of accumulating multiple data at the second granularity within the second preset time period before the current time period is used as the data summary result at the first granularity within the second preset time period before the current time period.

[0089] For example, for January 1, 2022, we obtained the sum of the total number of times application M was accessed from December 2, 2021 to December 31, 2021 (a total of 30 totals), and used this sum as the total number of times application M was accessed in the past 30 days before January 1, 2022.

[0090] In one embodiment of this specification, after step 309, the second-granularity data for all time periods can be output at once, and the task refresh is completed.

[0091] Subsequently, based on the UDTF (Split Calculation Component), the summary data for multiple days (such as the 365 days of the entire year of 2022 mentioned above) is split into daily summary data and inserted into the multi-day partition of the target table.

[0092] For example, taking the aforementioned data refresh application scenario as an example: In each time period of 2022, such as each day, the total number of times an application M was accessed in the past 30 days relative to that day is counted. Based on the above... Figure 3 The illustrated process, when calculating each day, avoids the problem of reading 29 days of data that are repeated from the previous day, thus preventing a significant waste of reading time. Furthermore, based on the above... Figure 3 The process shown, when calculating for each day, utilizes the same calculation data as the previous day. Therefore, the calculation results from the previous day can be referenced, thereby reducing the computational load. For example, the second granularity data of the first date used in the previous day can be subtracted from the previous day's calculation result, and the second granularity data of the last date required for the current day can be added, eliminating the need for extensive repetitive calculations. The more data accumulated each time, the more significant the reduction in computational load. For instance, in the calculation for each day, existing technologies require accumulating 10,000 data points to complete the statistics for that day, requiring 10,000 data point accumulations (9999 addition operations) for each day's calculation. However, using the method of the embodiments in this specification, in the calculation for the next day, only one data point needs to be subtracted and one data point added to the previous day's calculation result (one addition operation and one subtraction operation, a total of 2 operations). It is evident that compared to the 9999 addition operations of existing technologies, the computational load is significantly reduced, saving computational resources.

[0093] In one embodiment of this specification, a data refresh device is provided, see [link to relevant documentation]. Figure 4 The device includes:

[0094] Modify the code acquisition module 401 to obtain the modified calculation code; wherein, the modified calculation code corresponds to the data summary of the first granularity;

[0095] The granularity decomposition module 402 is configured to decompose a second granularity from a first granularity; wherein the second granularity is a finer granularity relative to the first granularity.

[0096] The data refresh task acquisition module 403 is configured to obtain a data refresh task based on the modified calculation code; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the corresponding first granularity data;

[0097] The fine-grained data acquisition module 404 is configured to acquire all the second-grained data required by the data refresh task according to the data refresh task.

[0098] The summary module 405 is configured to summarize data using all the acquired second-granularity data.

[0099] In one embodiment of the device described in this specification, the fine-grained data acquisition module 404 is configured to perform: read all the second-granularity data required for the data flushing task from the data warehouse in one go.

[0100] In one embodiment of the device described in this specification, the fine-grained data acquisition module 404 is configured to perform the following: before reading all the second-granularity data required for the data rollback task from the data warehouse at once, determine whether the data warehouse contains all the second-granularity data required for the data rollback task; if yes, continue to execute the step of reading all the second-granularity data required for the data rollback task from the data warehouse at once; if no, obtain the task code for acquiring the second-granularity data input by the administrator; run the task code for acquiring the second-granularity data to obtain all the second-granularity data required for the data rollback task.

[0101] In one embodiment of the device described in this specification, the data aggregation at the first granularity includes: summarizing the data aggregation results of the second preset time period prior to the current time period in each time cycle within a first preset time period.

[0102] In one embodiment of the device described in this specification, the summarizing module 405 is configured to perform:

[0103] In each time period within the first preset time period, multiple data points of the second granularity required in the current time period are extracted from all the read second granularity data using a sliding window method;

[0104] Using the extracted data of the second granularity required within the current time period, the result of summing up the data of the second granularity within the second preset time period before the current time period is calculated.

[0105] The result of summing up multiple data points of the second granularity within the second preset time period before the current time period is taken as the data summary result of the first granularity within the second preset time period before the current time period.

[0106] In one embodiment of the device described in this specification, the summarizing module 405 is configured to perform:

[0107] Subtract the second-granularity data for the first date required for the previous time period from the summary results of the previous time period, and add the second-granularity data for the last date required for the current time period.

[0108] In one embodiment of the device described in this specification, the data at the first granularity includes: the total number of visits to the designated platform across the entire network; correspondingly, the data at the second granularity includes: the number of visits to all merchants or all applications on the designated platform.

[0109] In one embodiment of the device described in this specification, the data at the first granularity includes: the total number of visits to the specified platform within a third preset time period; correspondingly, the data at the second granularity includes: the total number of visits to the specified platform within each fourth preset time period; wherein, the third preset time period is equal to the sum of multiple fourth preset time periods.

[0110] It should be noted that the above-mentioned devices are typically implemented on the server side. They can be set up on independent servers, or some or all of the devices can be combined and installed on the same server. This server can be a single server or a server cluster consisting of multiple servers. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. The above-mentioned devices can also be implemented on computer terminals with strong computing capabilities.

[0111] This specification provides, in one embodiment, a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods of any embodiment in the specification.

[0112] This specification provides a computing device according to one embodiment, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to perform the method of any embodiment of the specification.

[0113] It is understood that the structures illustrated in the embodiments of this specification do not constitute a specific limitation on the apparatus of the embodiments of this specification. In other embodiments of the specification, the above-described apparatus may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0115] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0116] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. Data rollback method, among which, The method includes: The modified calculation code is obtained; where the modified calculation code corresponds to the summary of the data at the first granularity. A second particle size is derived from the first particle size; wherein the second particle size is a finer particle size relative to the first particle size. Based on the modified calculation code, a data refresh task is obtained; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the data corresponding to the first granularity. Based on the data refresh task, obtain all the second-granularity data required by the data refresh task; In each time period, multiple data points of the second granularity required for the current time period are selected from all the acquired second-granularity data and the data is summarized. The aggregation of the first granularity of data includes: summarizing the data from the second preset time period before the current time period in each time period within the first preset time period; The step of selecting multiple data points of the second granularity required for the current time period from all acquired second-granularity data in each time period and summarizing the data includes: In each time period within the first preset time period, multiple data points of the second granularity required in the current time period are extracted from all the read second granularity data using a sliding window method; Using the extracted data of the second granularity required within the current time period, the result of summing up the data of the second granularity within the second preset time period before the current time period is calculated. The result of summing up multiple data points of the second granularity within the second preset time period before the current time period is taken as the data summary result of the first granularity within the second preset time period before the current time period.

2. The method according to claim 1, wherein, The process of obtaining all the second-granularity data required for the data rollback task includes: reading all the second-granularity data required for the data rollback task from the data warehouse in one go.

3. The method according to claim 2, wherein, Before reading all the second-granularity data required for the data flushing task from the data warehouse in one go, the process further includes: Determine whether the data warehouse contains all the second-granularity data required for the data rollback task; If so, continue with the step of reading all the second-granularity data required for the data flushing task from the data warehouse in one go; If not, obtain the task code for retrieving the second granularity data input by the administrator; run the task code to obtain all the second granularity data required by the data flushing task.

4. The method according to claim 1, wherein, The result of accumulating multiple data points of the second granularity within the second preset time period prior to the current time period includes: Subtract the second-granularity data for the first date required for the previous time period from the summary results of the previous time period, and add the second-granularity data for the last date required for the current time period.

5. The method according to claim 1, wherein, The data at the first granularity includes: the total number of visits to the specified platform across the entire network; correspondingly, the data at the second granularity includes: the number of visits to all merchants or all applications on the specified platform; or, The data at the first granularity includes: the total number of visits to the specified platform within a third preset time period; correspondingly, the data at the second granularity includes: the total number of visits to the specified platform within each fourth preset time period; wherein, the third preset time period is equal to the sum of multiple fourth preset time periods.

6. Data refresh device, wherein, The device includes: Modify the code acquisition module to obtain the modified calculation code; where the modified calculation code corresponds to the summary of the first granularity of data. The granularity decomposition module is configured to decompose a second granularity from a first granularity; wherein the second granularity is a finer granularity than the first granularity. The data refresh task acquisition module is configured to obtain a data refresh task based on the modified calculation code; wherein, the data refresh task includes: summarizing multiple data at the second granularity in order to obtain a summary of the corresponding first granularity data; The fine-grained data acquisition module is configured to acquire all second-granularity data required by the data refresh task based on the data refresh task. The aggregation module is configured to select multiple data points of the second granularity required for the current time period from all the acquired second-granularity data in each time period for data aggregation. The data aggregation at the first granularity includes: summarizing the data from the second preset time period prior to the current time period in each time cycle within the first preset time period; The aggregation module is configured to execute: In each time period within the first preset time period, multiple data points of the second granularity required in the current time period are extracted from all the read second granularity data using a sliding window method; Using the extracted data of the second granularity required within the current time period, the result of summing up the data of the second granularity within the second preset time period before the current time period is calculated. The result of summing up multiple data points of the second granularity within the second preset time period before the current time period is taken as the data summary result of the first granularity within the second preset time period before the current time period.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-5.

8. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-5.

Images