Method and device for updating big data calculation results with resistance to volatility

Abstract

The embodiment of the present invention discloses a method and device for updating big data calculation results with resistance to volatility, the method comprising: obtaining consumption data to be calculated based on the Doris database; aggregating and calculating the consumption data to be calculated according to the time interval to obtain first calculation data; writing the first calculation data into the first temporary partition; correcting the first calculation data to obtain second calculation data; writing the second calculation data into the second temporary partition; replacing the second temporary partition with the formal partition of the corresponding time interval; reading the calculation data in the corresponding formal partition, the calculation data including the second calculation data, aggregating and calculating to form a big data calculation result report with a version. It can be used to solve the problem of poor user experience caused by large volatility of data update.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method and device for updating big data calculation results with resistance to fluctuations. Background Art

[0002] When the project team provides API reports to advertisers or channel suppliers, the report data involves dynamic advertising attribution, so historical data needs to be updated frequently (for example, retrospectively calculated every 15 minutes), including deletion and modification. The calculation process involves ultra-large-scale big data calculations, so there is often a problem of large volatility in big data that affects customer experience.

[0003] The large-scale API data of foreign advertising companies also have the problem of big data volatility, but the companies will explain that the data during the process is only for reference, and the final data is based on the billing data. As for the principle of volatility, let's take an example below. Assuming that the advertising revenue is settled on a daily basis, the following table is a week's data.

[0004] On 2024-08-29, the cost of the ads seen by advertisers was consumed (Table 1):

[0005] Table 1 20240829 report

[0006]

[0007] On August 30, 2024, due to changes in advertising attribution, the CDE settlement item was deleted, and then the KOL settlement item was added. The account you see at this time is Table 2:

[0008] Table 2 20240830 Report

[0009]

[0010] If traditional technology is used, the KOL settlement item is inserted first. When the advertisement user queries the item right after the insertion, he will find that the consumption on 2024-08-29 is always 100 + 200 + 300 + 100 + 300 + 50 = 1050 (process value). If the CDE settlement item is deleted at this time, the user will query the item at this time, and the displayed settlement consumption is: 100 + 200 + 300 + 100 + 300 + 50 - 200 = 850 (stable value). This is just the simplest example, but the user has seen two values ​​1050 and 850 respectively. In this way, the data fluctuates. If the data volume is small and the process is fast, the user may not feel it. However, if the data is massive, such as the update time is more than 30 minutes, the fluctuating data will have a poor user experience. If a worse technical solution is adopted, such as deleting all data on the 29th first and then adding data on the 29th; or adding data on the 29th first and then deleting the redundant data on the 29th, the data displayed to the user may be a continuous change from 0 to a stable value or from a process value to a stable value, which will give the user a worse experience. Moreover, the data for advertising attribution is massive, which requires a higher stability for data updates.

[0011] How to prevent users from perceiving the extreme changes of big data during the data update process is a key factor in improving user experience. Therefore, in view of the current problem of poor user experience caused by large volatility of data updates, it is necessary to conduct further research on big data result update methods that resist volatility. Summary of the invention

[0012] The embodiments of the present invention provide a method and device for updating big data calculation results that are resistant to volatility, so as to solve the problem that the current data update volatility is large and leads to poor user experience.

[0013] In one aspect, the present invention provides a method for updating big data calculation results with resistance to volatility, comprising:

[0014] Obtain the consumption data to be calculated based on the doris database;

[0015] Aggregating and calculating the consumption data to be calculated according to the time interval to obtain first calculation data;

[0016] Writing the first calculated data into the first temporary partition;

[0017] Performing data correction on the first calculation data to obtain second calculation data;

[0018] Writing the second calculated data into the second temporary partition;

[0019] Replace the second temporary partition with the formal partition of the corresponding time interval;

[0020] The calculation data in the corresponding formal partition is read, the calculation data includes the second calculation data, and the calculation is aggregated to form a versioned big data calculation result report.

[0021] Furthermore, the step of performing data correction on the first calculation data to obtain the second calculation data includes:

[0022] Write the first calculated data into the mount table of the doris database;

[0023] Based on the mount table, data correction is performed on the first calculation data to obtain second calculation data.

[0024] Furthermore, the mounted table is a MySQL database, and the MySQL database is an external table of the doris database.

[0025] Furthermore, before the step of obtaining the consumption data to be calculated based on the doris database, the method further includes:

[0026] Use the Kafka data storage platform to asynchronously connect the original data to the doris database to form the consumption data to be calculated; or use the insert into select x from x statement to import the original data from the doris database to form the consumption data to be calculated.

[0027] Furthermore, the steps of using the Kafka data storage platform to asynchronously connect the original data to the doris database to form the consumption data to be calculated include:

[0028] Write the original data into the Kafka data storage platform;

[0029] By using the Routine load capability of the doris database, the original data is written into the table to be calculated table_a within the doris database to form the consumption data to be calculated.

[0030] Furthermore, the step of aggregating and calculating the consumption data to be calculated according to the time interval to obtain the first calculation data includes:

[0031] Based on the table to be calculated table_a, the corresponding consumption data to be calculated are aggregated and calculated according to the time interval to obtain the first calculated data;

[0032] The step of writing the first calculated data into the first temporary partition includes:

[0033] The first calculated data is written into the first temporary partition of the result table table_b.

[0034] Furthermore, the time interval is hours, and the table to be calculated table_a adopts hourly partitioning technology.

[0035] Furthermore, the result table table_b also includes a second temporary partition, and the step of replacing the second temporary partition with a formal partition of the corresponding time interval includes:

[0036] The second temporary partition is named the formal partition according to the corresponding time interval.

[0037] Further, the calculation data in the corresponding formal partition is read, the calculation data includes the second calculation data, and the steps of aggregating and calculating and forming a big data calculation result report with a version include:

[0038] Read the calculation data of multiple corresponding formal partitions through the aggregate calculation API server;

[0039] Aggregate and calculate the data to obtain the calculation results;

[0040] Generating a big data calculation result report according to the calculation result, wherein the version number of the big data calculation result report is determined according to the corresponding time interval of the second calculation data;

[0041] Real-time display of big data calculation result reports.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] By storing the data of the big data calculation process in the temporary partition of the database, and the report calls the data of the formal partition that has been processed, and the calculation results displayed to customers are accompanied by version information, stable and quasi-real-time data are provided to advertisers or suppliers. The old plan and the hundreds of transaction data differences every day will no longer appear, which solves the current problem of poor user experience caused by large volatility of data updates.

[0044] On the other hand, the present invention provides a device for updating a big data calculation result with resistance to fluctuation, which is used to implement the above-mentioned method for updating a big data calculation result with resistance to fluctuation, comprising:

[0045] The acquisition module is used to obtain the consumption data to be calculated based on the doris database;

[0046] An aggregation module, used for aggregating and calculating the consumption data to be calculated according to a time interval to obtain first calculation data;

[0047] A first writing module, used for writing the first calculation data into the first temporary partition;

[0048] A correction module, used for performing data correction on the first calculation data to obtain second calculation data;

[0049] A second writing module, used for writing the second calculated data into the second temporary partition;

[0050] A partition replacement module, used to replace the second temporary partition with a formal partition of a corresponding time interval;

[0051] The calculation module is used to read the calculation data in the corresponding formal partition, the calculation data includes the second calculation data, aggregate the calculation and form a versioned big data calculation result report. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0053] Figure 1 It is a flowchart of a method for updating big data calculation results with resistance to volatility according to an embodiment of the present invention.

[0054] Figure 2 It is a schematic diagram of a big data calculation result updating device with anti-volatility according to another embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 A method for updating big data calculation results with resistance to volatility according to an embodiment of the present invention comprises the following steps:

[0057] Step 11, obtain the consumption data to be calculated based on the doris database.

[0058] In the embodiment of the present invention, the Doris database has the function of temporary partitioning. During calculation, data can be written into the temporary partition, and the data of the temporary partition will not be displayed in the report. Therefore, the Doris database is used to obtain the consumption data to be calculated for further data processing.

[0059] Among them, as one implementation of the embodiment of the present invention, before step 11, it also includes:

[0060] Use the Kafka data storage platform to asynchronously connect the original data to the doris database to form the consumption data to be calculated; or use the insert into select x from x statement to import the original data from the doris database to form the consumption data to be calculated.

[0061] Specifically, the steps of using the Kafka data storage platform to asynchronously connect the original data to the doris database to form the consumption data to be calculated include:

[0062] Write the original data into the Kafka data storage platform;

[0063] By using the Routine load capability of the doris database, the original data is written into the table to be calculated table_a within the doris database to form the consumption data to be calculated.

[0064] In the embodiment of the present invention, the original data access method is involved. Due to the large amount of data, all data is first written to Kafka, and then the Routine load capability provided by doris is used to write the data to the original table table_a inside doris.

[0065] Step 12: Aggregate and calculate the consumption data to be calculated according to the time interval to obtain first calculation data.

[0066] In the embodiment of the present invention, the consumption data to be calculated may be aggregated and calculated according to time intervals. As one implementation method, step 12 may include:

[0067] Based on the table to be calculated table_a, the corresponding consumption data to be calculated are aggregated and calculated according to the time interval to obtain the first calculated data.

[0068] In the embodiment of the present invention, the time interval can be set to hours, and the table to be calculated table_a adopts the hourly partitioning technology. The table to be calculated table_a adopts the hourly partitioning technology, that is, the data with the same transaction date + hour will be put together, and then the data will be aggregated and calculated at the hourly level. That is, when calculating big data, the data will be aggregated and calculated for each day every hour, and then the results will be directly inserted into the first temporary partition (this is a pure SQL calculation).

[0069] Step 13: write the first calculated data into the first temporary partition.

[0070] In the embodiment of the present invention, in order to ensure the anti-volatility of the update of the big data calculation result, the first calculation data of the calculation process is written into one of the temporary partitions, which can be the first temporary partition.

[0071] Specifically, step 13 may include:

[0072] The first calculated data is written into the first temporary partition of the result table table_b.

[0073] That is, the doris database includes the table to be calculated table_a and the result table table_b. After data aggregation and calculation are performed at the hourly level in the table to be calculated table_a, the aggregated data is saved in the first temporary partition of the result table table_b. The data in the first temporary partition is process data and is not included in the calculation results displayed to customers. It can be used to achieve the anti-volatility of updating big data calculation results.

[0074] Step 14: perform data correction on the first calculation data to obtain second calculation data.

[0075] In the embodiment of the present invention, the first calculated data is obtained by pure SQL aggregation calculation, and data correction is required. Specifically, step 14 includes:

[0076] Write the first calculated data into the mount table of the doris database;

[0077] Based on the mount table, data correction is performed on the first calculation data to obtain second calculation data.

[0078] In the embodiment of the present invention, the mount table is a MySQL database, and the MySQL database is an external table of the doris database.

[0079] In the embodiment of the present invention, after the aggregated data is saved in the first temporary partition of the result table table_b, the data needs to be corrected, but the correction logic is very complicated and cannot be aggregated and calculated by SQL, so it needs to be read out and put into other systems (such as Python program system) for calculation. After the calculation, the data needs to be written back to the doris database, but doris does not recommend using insert into to write directly, so the data is first written to the mysql database. The mysql database is an external table of doris and can be used directly in doris.

[0080] Step 15: Write the second calculated data into the second temporary partition.

[0081] In this step, the corrected second calculation data may be written into another temporary partition of the result table table_b, namely, the second temporary partition, to facilitate subsequent operations.

[0082] Step 16: Replace the second temporary partition with the formal partition of the corresponding time interval.

[0083] In the embodiment of the present invention, the modified second calculation data needs to be applied to the result report, so the second temporary partition needs to be replaced by a formal partition of the corresponding time interval.

[0084] Step 16 may specifically include:

[0085] The second temporary partition is named the formal partition according to the corresponding time interval.

[0086] For example, when the time interval corresponds to 0:00 on August 19, 2024, you can point 2024-08-19_00 to the second temporary partition (the second temporary partition will no longer exist), ultimately achieving the effect of stable data update.

[0087] Step 17, reading the calculation data in the corresponding formal partition, the calculation data including the second calculation data, performing aggregation calculation and forming a versioned big data calculation result report.

[0088] In the embodiment of the present invention, because the entire calculation process is carried out in the temporary partition, only after the calculation is completed, the data of the hourly partition will be pointed to the temporary partition (partition replacement is equivalent to changing the name, and the temporary partition will no longer exist), and then the program API reads the data calculated in the formal partition where the calculation is completed, so the data reaches a certain state.

[0089] In the embodiment of the present invention, step 17 may include:

[0090] Read the calculation data of multiple corresponding formal partitions through the aggregate calculation API server;

[0091] Aggregate and calculate the data to obtain the calculation results;

[0092] Generating a big data calculation result report according to the calculation result, wherein the version number of the big data calculation result report is determined according to the corresponding time interval of the second calculation data;

[0093] Real-time display of big data calculation result reports.

[0094] In the process of updating big data, especially when the data volume is very large, the update time is generally very long. During the update process, the values ​​will constantly change. This change process is perceived by the user and has a great impact on the user experience. If the implementation method of the embodiment of the present invention is adopted, after the calculation is completed, the latest version of the result (with the version number) is provided to the user for viewing.

[0095] For example, when calculating version 3 data, if it is not completed, the user can see data that is not version 3. At this time, they can only see version 2 data. After completion, they can see version 3 data. In this way, no matter how drastically the version 3 calculation process changes, the user will not be aware of it. The implementation method of the embodiment of the present invention applies the temporary partition function in the Doris database. During the calculation, the data can be written into the temporary partition. When the calculation is completed, the temporary partition is named as the formal partition. In this way, the user cannot perceive the change of the data, which solves the problem of drastic data fluctuations. However, when Doris inserts a large amount of data, it is not recommended to use a third-party program to directly insert into to import external data. It is recommended to use Kafka asynchronous access, or use insert in to select x from x statement to import data from the inside, so the technical framework has been further optimized.

[0096] Therefore, compared with the prior art, the present invention has the following advantages:

[0097] By storing the data of the big data calculation process in the temporary partition of the database, and the report calls the data of the formal partition that has been processed, and the calculation results displayed to customers are accompanied by version information, stable and quasi-real-time data are provided to advertisers or suppliers. The old plan and the hundreds of transaction data differences every day will no longer appear, which solves the current problem of poor user experience caused by large volatility of data updates.

[0098] See also Figure 2 According to another embodiment of the present invention, a device 20 for updating big data calculation results with resistance to fluctuations can be used to implement the corresponding steps of the above method embodiment, including:

[0099] The acquisition module 21 is used to acquire the consumption data to be calculated based on the doris database.

[0100] The aggregation module 22 is used to aggregate and calculate the consumption data to be calculated according to the time interval to obtain first calculation data.

[0101] The first writing module 23 is used to write the first calculated data into the first temporary partition.

[0102] The correction module 24 is used to correct the first calculation data to obtain second calculation data.

[0103] The second writing module 25 is used to write the second calculated data into the second temporary partition.

[0104] The partition replacement module 26 is used to replace the second temporary partition with a formal partition of the corresponding time interval.

[0105] The calculation module 27 is used to read the calculation data in the corresponding formal partition, the calculation data includes the second calculation data, aggregate the calculation and form a versioned big data calculation result report.

[0106] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for updating big data calculation results with resistance to volatility, characterized in that: include: Obtain the consumption data to be calculated based on the doris database; Aggregating and calculating the consumption data to be calculated according to the time interval to obtain first calculation data; Writing the first calculated data into a first temporary partition; Performing data correction on the first calculation data to obtain second calculation data; Writing the second calculated data into a second temporary partition; Replacing the second temporary partition with a formal partition of the corresponding time interval; Read the calculation data in the corresponding formal partition, the calculation data including the second calculation data, perform aggregation calculation and form a versioned big data calculation result report; The doris database includes a table to be calculated table_a, and the step of aggregating and calculating the consumption data to be calculated according to the time interval to obtain the first calculation data includes: Based on the table to be calculated table_a, the corresponding consumption data to be calculated are aggregated and calculated according to the time interval to obtain the first calculated data; The step of writing the first calculated data into the first temporary partition comprises: Write the first calculated data into the first temporary partition of the result table table_b; The result table table_b also includes the second temporary partition, and the step of replacing the second temporary partition with the formal partition of the corresponding time interval includes: Name the second temporary partition as a formal partition according to the corresponding time interval; The step of performing data correction on the first calculation data to obtain second calculation data includes: Writing the first calculation data into the mount table of the doris database; Based on the mount table, performing data correction on the first calculation data to obtain the second calculation data; The step of reading the calculation data in the corresponding formal partition, wherein the calculation data includes the second calculation data, performing aggregation calculations and forming a versioned big data calculation result report, comprises: Read the calculation data of multiple corresponding formal partitions through the aggregate calculation API server; Aggregate and calculate the calculation data to obtain a calculation result; forming the big data calculation result report according to the calculation result, wherein the version number of the big data calculation result report is determined according to the corresponding time interval of the second calculation data; The big data calculation result report is displayed in real time.

2. The method for updating big data calculation results with resistance to volatility according to claim 1, characterized in that: The mount table is a MySQL database, and the MySQL database is an external table of the doris database.

3. The method for updating big data calculation results with resistance to volatility according to claim 1, characterized in that: Before the step of obtaining the consumption data to be calculated based on the doris database, the method further includes: The original data is asynchronously connected to the doris database using the Kafka data storage platform to form the consumption data to be calculated; or, the original data is imported from the doris database using the insert into select x from x statement to form the consumption data to be calculated.

4. The method for updating big data calculation results with resistance to volatility according to claim 3 is characterized in that: The step of using the Kafka data storage platform to asynchronously access the original data to the doris database to form the consumption data to be calculated includes: Writing the original data into the Kafka data storage platform; By utilizing the Routine load capability of the doris database, the original data is written into the table to be calculated table_a inside the doris database to form the consumption data to be calculated.

5. The method for updating big data calculation results with resistance to volatility according to claim 4, characterized in that: The time interval is hours, and the table to be calculated table_a adopts hourly partitioning technology.

6. A device for updating big data calculation results with resistance to fluctuations, characterized in that: The anti-volatility big data calculation result updating device is used to implement the anti-volatility big data calculation result updating method according to any one of claims 1 to 5, and the anti-volatility big data calculation result updating device comprises: The acquisition module is used to obtain the consumption data to be calculated based on the doris database; an aggregation module, configured to aggregate and calculate the consumption data to be calculated according to a time interval to obtain first calculation data; A first writing module, used for writing the first calculated data into a first temporary partition; A correction module, used for performing data correction on the first calculation data to obtain second calculation data; A second writing module, used for writing the second calculated data into a second temporary partition; A partition replacement module, used to replace the second temporary partition with a formal partition of a corresponding time interval; A calculation module, used to read the calculation data in the corresponding formal partition, the calculation data including the second calculation data, perform aggregation calculation and form a versioned big data calculation result report; The doris database includes a table to be calculated table_a, and the aggregation module is further used to: based on the table to be calculated table_a, perform data aggregation and calculation on the corresponding consumption data to be calculated according to the time interval to obtain the first calculation data; The first writing module is further used to: write the first calculated data into the first temporary partition of the result table table_b; The partition replacement module is further used to: name the second temporary partition as a formal partition according to the corresponding time interval; The correction module is also used to: write the first calculation data into the mount table of the doris database; based on the mount table, perform data correction on the first calculation data to obtain the second calculation data; The reading module is also used to: read the calculation data of multiple corresponding formal partitions through the aggregate calculation API server; aggregate the calculation data to obtain the calculation results; form the big data calculation result report according to the calculation results, and the version number of the big data calculation result report is determined according to the corresponding time interval of the second calculation data; and display the big data calculation result report in real time.

Images