Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: when a calculation task corresponding to a statistical time window obtains target data, obtaining aggregation data corresponding to an object identifier from an aggregation index data table based on the object identifier corresponding to the target data, the aggregated data comprises water level data corresponding to the calculation task, an aggregated index value and a consumed data set corresponding to the calculation task; determining the validity of the target data according to the water level line data and the consumed data set; and under the condition that the target data is valid, executing the calculation task based on the aggregation index value and the target data, and updating the aggregation index value in the aggregation index data table according to a calculation result. By adopting the method, the statistical efficiency and the real-time performance of the aggregation indexes can be improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] When conducting user profiling, marketing activities, and report analysis in cloud service products or the Internet, the typical scenario is to obtain the aggregated indicators of users over a period of time in real time. For example, in the payment and billing scenario, it is often necessary to count the total amount of users' cumulative consumption in recent years, and then use this indicator as the basis for judging VIP (very important person) users and conducting targeted marketing activities.

[0003] However, when the statistical time window spans a large amount, it often involves massive storage data, which will cause the calculation of aggregate indicators to take a long time, resulting in low statistical efficiency of aggregate indicators, and thus making it impossible to provide efficient, timely and accurate responses for product marketing. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the statistical efficiency and accuracy of aggregate indicators in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] When the computing task corresponding to the statistical time window obtains the target data, based on the object identifier corresponding to the target data, the aggregated data corresponding to the object identifier is obtained from the aggregated indicator data table, the aggregated data including the watermark data corresponding to the computing task, the aggregated indicator value, and the consumed data set corresponding to the computing task;

[0007] Determining the validity of the target data according to the waterline data and the consumed data set;

[0008] When the target data is valid, the calculation task is performed based on the aggregation index value and the target data, and the aggregation index value in the aggregation index data table is updated according to the calculation result.

[0009] In one embodiment, the computing task corresponding to the statistical time window includes an accumulation task and a subtraction task.

[0010] The waterline data corresponding to the accumulation task is the storage time of the right boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the accumulation task is the set of the right boundary data of the statistical time window that has been consumed;

[0011] The waterline data corresponding to the subtraction task is the storage time of the left boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the subtraction task is the set of the left boundary data of the statistical time window that has been consumed.

[0012] In one embodiment, the computing task is the accumulation task, and determining the validity of the target data according to the waterline data and the consumed data set includes:

[0013] Obtaining the storage time and storage identification of the target data;

[0014] If the storage time is greater than the waterline data corresponding to the accumulation task, and the storage identifier is not recorded in the consumed data set corresponding to the accumulation task, it is determined that the target data is valid.

[0015] In one embodiment, the calculation task is the subtraction task, and determining the validity of the target data according to the waterline data and the consumed data set includes:

[0016] Obtaining the storage time and storage identification of the target data;

[0017] If the warehousing time is greater than the difference between the current calculation time and the window length of the time window, and the warehousing time is greater than the waterline data corresponding to the subtraction task, and the warehousing identifier is not recorded in the consumed data set corresponding to the subtraction task, then the target data is determined to be valid.

[0018] In one of the embodiments, after updating the aggregation index value in the aggregation index data table according to the calculation result, the method further includes:

[0019] According to the storage time corresponding to the target data, update the waterline data corresponding to the computing task included in the aggregated data in the aggregated indicator data;

[0020] The storage identification corresponding to the target data is added to the consumed data set corresponding to the computing task included in the aggregated data in the aggregated indicator data.

[0021] In one embodiment, the statistical time window has a left boundary and a right boundary, the left boundary corresponds to a subtraction task, and the right boundary corresponds to an accumulation task, and the method further includes:

[0022] Slide the statistical time window according to a preset time step;

[0023] Based on the sliding of the left boundary and / or the right boundary of the statistical time window, data is scanned in the basic data table, and the scanned data is used as the target data. The basic data table includes an entry time field, and the entry time corresponding to the target data is within the sliding range of the left boundary and / or the right boundary.

[0024] In a second aspect, the present application further provides a data processing device, comprising:

[0025] An acquisition module, configured to acquire, when the computing task corresponding to the statistical time window acquires the target data, aggregate data corresponding to the object identifier from the aggregate indicator data table based on the object identifier corresponding to the target data, wherein the aggregate data includes waterline data corresponding to the computing task, an aggregate indicator value, and a consumed data set corresponding to the computing task;

[0026] A determination module, configured to determine the validity of the target data according to the waterline data and the consumed data set;

[0027] A processing module is used to perform the calculation task based on the aggregation index value and the target data when the target data is valid, and to update the aggregation index value in the aggregation index data table according to the calculation result.

[0028] In one embodiment, the computing task corresponding to the statistical time window includes an accumulation task and a subtraction task.

[0029] The waterline data corresponding to the accumulation task is the storage time of the right boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the accumulation task is the set of the right boundary data of the statistical time window that has been consumed;

[0030] The waterline data corresponding to the subtraction task is the storage time of the left boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the subtraction task is the set of the left boundary data of the statistical time window that has been consumed.

[0031] In one embodiment, the calculation task is the accumulation task, and the determination module is specifically used for:

[0032] Obtaining the storage time and storage identification of the target data;

[0033] If the storage time is greater than the waterline data corresponding to the accumulation task, and the storage identifier is not recorded in the consumed data set corresponding to the accumulation task, it is determined that the target data is valid.

[0034] In one embodiment, the calculation task is the subtraction task, and the determination module is specifically used for:

[0035] Obtaining the storage time and storage identification of the target data;

[0036] If the warehousing time is greater than the difference between the current calculation time and the window length of the time window, and the warehousing time is greater than the waterline data corresponding to the subtraction task, and the warehousing identifier is not recorded in the consumed data set corresponding to the subtraction task, then the target data is determined to be valid.

[0037] In one of the embodiments, after the aggregation index value in the aggregation index data table is updated according to the calculation result, the device further includes an updating module for:

[0038] According to the storage time corresponding to the target data, update the waterline data corresponding to the computing task included in the aggregated data in the aggregated indicator data;

[0039] The storage identification corresponding to the target data is added to the consumed data set corresponding to the computing task included in the aggregated data in the aggregated indicator data.

[0040] In one embodiment, the statistical time window has a left boundary and a right boundary, the left boundary corresponds to a subtraction task, and the right boundary corresponds to an accumulation task, and the device further includes a scanning module for:

[0041] Slide the statistical time window according to a preset time step;

[0042] Based on the sliding of the left boundary and / or the right boundary of the statistical time window, data is scanned in the basic data table, and the scanned data is used as the target data. The basic data table includes an entry time field, and the entry time corresponding to the target data is within the sliding range of the left boundary and / or the right boundary.

[0043] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above data processing methods when executing the computer program.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above data processing methods when executed by a processor.

[0045] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements any of the data processing methods described above when executed by a processor.

[0046] The above-mentioned data processing method, apparatus, computer device, computer-readable storage medium and computer program product, when the computing task corresponding to the statistical time window obtains the target data, obtains the aggregated data corresponding to the object identifier from the aggregated indicator data table based on the object identifier corresponding to the target data, and the aggregated data includes the waterline data corresponding to the computing task, the aggregated indicator value, and the consumed data set corresponding to the computing task. The validity of the target data is determined based on the waterline data and the consumed data set, and when the target data is valid, the computing task is executed based on the aggregated indicator value and the target data, and the aggregated indicator value in the aggregated indicator data table is updated according to the calculation result. The data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product provided by the embodiments of the present disclosure store various types of aggregate index values ​​corresponding to each object through an aggregate index data table, and obtain new data through real-time scanning of a statistical time window to update the aggregate index value in real time, that is, the aggregate index value is updated based on real-time data, so the amount of calculated data is small, which greatly improves the statistical efficiency and real-time performance of the aggregate index, and during the update process, the validity of the acquired data is determined through waterline data and the consumed data set, and the aggregate index value is updated only based on valid data, so that interference from duplicate data and data that is not within the statistical period can be avoided, which can greatly improve the accuracy of the aggregate index value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is a schematic flow chart of a data processing method in one embodiment;

[0049] FIG2 is a schematic diagram of a statistical time window in one embodiment;

[0050] FIG3 is a schematic diagram of a sliding statistical time window in one embodiment;

[0051] FIG4 is a schematic diagram of a process for obtaining target data in one embodiment;

[0052] FIG5 is a schematic diagram of a flow chart of step 104 in one embodiment;

[0053] FIG6 is a schematic diagram of a flow chart of step 104 in one embodiment;

[0054] FIG7 is a schematic diagram of a data processing method in one embodiment;

[0055] FIG8 is a schematic diagram of a data processing method in one embodiment;

[0056] FIG9 is a schematic diagram of a data processing method in one embodiment;

[0057] FIG10 is a block diagram of a data processing device in one embodiment;

[0058] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In related technologies, the following three methods are often used when calculating aggregate indicators:

[0061] Method 1) Direct statistical access. In the process of direct statistical access, when it is necessary to obtain the cumulative value of a user in the last N years, the business application directly accesses the database to collect statistical data and obtain the result. Although this method is simple and easy to implement, and can directly obtain accurate indicator results, it is usually difficult to guarantee efficiency. In addition, the database resources are prone to computing bottlenecks when there are multiple concurrencies, resulting in unstable services, database memory overflow or stuck problems, and business is affected. In addition, access and calculation are carried out at the same time, with high coupling and no separation. All calculations rely on the database, and resources such as IO (Input Output Stream) and CPU (Central Processing Unit) on the database storage side are prone to bottlenecks. In other words, the direct statistical algorithm is relatively simple to implement, but the service is easily affected by computing power issues, and real-time performance cannot be guaranteed, especially in high-concurrency scenarios, and is not used for online real-time data services.

[0062] Method 2) Regular offline calculation. In the process of regular offline calculation, the cumulative values ​​of all users in the last N years can be calculated in advance and cached. The business side can read the cache in real time, and it can also support high-concurrency real-time response. The calculation side performs offline statistics regularly (for example, every day) and then updates the latest indicator value of each user. Although this method can support real-time high-concurrency reading by reading the cache value, and the business access result indicator and the calculation of the indicator are separated and decoupled, there is no mutual interference problem. However, the offline generation of aggregate indicators has a large delay and cannot quickly obtain the latest results. In addition, the cache is used to solve the problem of high-performance access, and the risk of cache data loss is high in the event of a failure. In addition, the cache is maintained as a high-concurrency OLTP () access, and the management cost of maintaining the cache component alone is high. That is, the solution of regular offline calculation and cache results solves the problem of high-concurrency access of direct statistics, but the risk of cache data loss in the event of a failure is high, and the management cost of maintaining the cache component alone is high. In addition, the statistical calculation is offline, and the latest value cannot be obtained, which makes it difficult to meet real-time requirements.

[0063] Method 3) Introducing real-time framework computing. Real-time computing is performed by introducing a third-party external computing framework, such as Flink. The real-time computing capabilities of the third-party framework can also quickly calculate relevant calculation results and store them in a relational database, so that indicator values ​​can be quickly calculated to meet real-time requirements. However, the third-party computing framework cluster needs to be maintained separately, which has high hardware and maintenance costs and high learning and technical barriers.

[0064] The disclosed embodiment provides a data processing method, which stores various types of aggregation indicator values ​​corresponding to each object through an aggregation indicator data table, and obtains new data through real-time scanning of a statistical time window to update the aggregation indicator value in real time, that is, the aggregation indicator value is updated based on real-time data, so the amount of calculated data is small, which greatly improves the statistical efficiency and real-time performance of the aggregation indicator, and in the updating process, the validity of the acquired data is judged through waterline data and consumed data sets, and the aggregation indicator value is updated only based on valid data, so that interference from duplicate data and data that is not within the statistical period can be avoided, which can greatly improve the accuracy of the aggregation indicator value.

[0065] In an exemplary embodiment, as shown in FIG1 , a data processing method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 102 to 106, wherein:

[0066] Step 102, when the computing task corresponding to the statistical time window obtains the target data, the aggregate data corresponding to the object identifier is obtained from the aggregate indicator data table based on the object identifier corresponding to the target data. The aggregate data includes the waterline data corresponding to the computing task, the aggregate indicator value, and the consumed data set corresponding to the computing task.

[0067] In the embodiments of the present disclosure, the aggregation index value corresponding to the target aggregation index is data related to a statistical period. In order to better achieve efficient calculation in the embodiments of the present disclosure, this statistical time period is defined as a sliding statistical time window. The statistical time window is a sliding window set for the target aggregation index. The window length of the statistical time window is the statistical period corresponding to the target aggregation index. For example, in the payment and billing scenario, the target aggregation index is the total consumption amount index, and the statistical period is one year, then the length of the statistical time window is 1 year. The sliding step size of the statistical time window is set by those skilled in the art as needed, and is not limited in the embodiments of the present disclosure.

[0068] For example, when a user generates new data, the new data will be written into the basic data table. As the statistical time window slides, the time range corresponding to the statistical time window will change. The new data entering the statistical time window can then be obtained from the basic data table and used as target data to participate in the calculation task corresponding to the statistical time window.

[0069] FIG2 shows a statistical time window with a time span (statistical period) of N, where the time unit of N can be hours, days, months, years, etc. The length of the statistical period within the last N time is determined by the start time T1 (left boundary) and the end time T2 (right boundary) of the statistical interval, and the time length of the window N=T2-T1. The embodiment of the present disclosure requires that N always remains the length of the statistical period, that is, the interval between the left and right boundaries is fixed.

[0070] Figure 3 shows the sliding process of the statistical time window. Assuming that the sliding step of the statistical time window is set to a fixed forward time frequency DeltaT, the left boundary T1 of the statistical time window slides forward in real time to become T3 as time changes, and the right boundary T2 of the statistical time window slides forward in real time to become T4 as time changes. The length of the statistical time window is fixed N, so it will pull the entire statistical time window forward and always keep T4-T3=N. The statistical time window will continue to slide forward with this frequency, and the forward pace frequency can be set to different paces as needed to control the frequency of the calculation program scanning data.

[0071] In an exemplary embodiment, the statistical time window has a left boundary and a right boundary, the left boundary corresponds to the subtraction task, and the right boundary corresponds to the accumulation task. As shown in FIG. 4 , the method may further include the following steps 402 to 404, wherein:

[0072] Step 402, sliding the statistical time window according to a preset time step;

[0073] Step 404, based on the sliding of the left boundary and / or the right boundary of the statistical time window, scan the data in the basic data table, and use the scanned data as the target data. The basic data table includes a storage time field, and the storage time corresponding to the target data is within the sliding range of the left boundary and / or the right boundary.

[0074] In the disclosed embodiment, the statistical time window slides according to the preset time step, and the data can be scanned in the basic data table based on the sliding of the left boundary and / or the sliding of the right boundary of the statistical time window. Exemplarily, assuming that the preset time step is 3s, the left boundary of the statistical time window is T1, and the right boundary is T2, the sliding trajectory of the left boundary is [T1, T1+3s], and the sliding trajectory of the right boundary is [T2, T2+3s]. Based on the sliding trajectories of the left and right boundaries, data is scanned from the basic data table, and the scanned data is used as the target data, and is respectively involved in the calculation of the target aggregation index corresponding to the left and right boundaries.

[0075] In the disclosed embodiment, a basic data table (such as a basic consumer behavior data table) can be designed in advance, and the characteristics of a relational database can be used to solve the problems of data disorder and duplication to generate a basic data table. This includes: (1) designing a reasonable business primary key according to the characteristics of the business, and using the characteristics of the primary key to deduplicate duplicate data to ensure that there is only one identical data entry. (2) When designing the table structure, an automatically generated entry time (create_time timestamp, hereinafter referred to as CT) is set for each row of data with an accuracy of at least microseconds. The purpose is to ensure that different rows of data entered into the database are incremented in order according to the CT field and that the amount of data at the same time is as small as possible. The acquisition module writes the data serially into a distributed relational database to generate a basic data table for subsequent calculations, data verification, and monitoring. The use of a distributed database not only solves the storage of basic massive data, but also ensures efficient read and write performance.

[0076] Online business applications generate consumption-related behavior data or order data at all times. Applications can write consumption behavior data to distributed databases in real time. If the application does not support direct writing, the collection and storage of relevant data can be completed through the collection program. In addition, if there is historical base data, the import of historical data can also be completed in advance through the collection program.

[0077] After obtaining the target data, the aggregated data corresponding to the object identifier can be obtained from the aggregated indicator data table based on the object identifier of the target data. The object identifier of the target data is used to uniquely identify the object to which the target data belongs, such as user ID, product ID and other identification information.

[0078] The aggregate indicator value of each object for the aggregate indicator can be recorded in the aggregate indicator data table. In one example, the aggregate indicator data table can only record the aggregate indicator value corresponding to the target aggregate indicator, so after obtaining the target data, the corresponding aggregate indicator value can be obtained from the aggregate data table based on the object identifier corresponding to the target data, and the aggregate indicator value is the indicator value of the target aggregate indicator calculated last time. In another example, the aggregate indicator value corresponding to each aggregate indicator can be recorded in the aggregate indicator data table, so after obtaining the target data, the corresponding aggregate indicator value can be obtained from the aggregate data table based on the object identifier corresponding to the target data and the indicator identifier of the target aggregate indicator. After obtaining the aggregate indicator value, the aggregate indicator value can be updated based on the target data.

[0079] The aggregate index data table may also include waterline data and a consumed data set, wherein the waterline data is the storage time of the latest consumed data, and the consumed data set includes the id information of the data that has participated in the calculation of the aggregate index.

[0080] When the target data is obtained, in one example, the corresponding waterline data and the consumed data set can also be obtained from the aggregate indicator data table based on the object identifier corresponding to the target data and the indicator identifier of the target aggregation indicator; in another example, the corresponding waterline data and the consumed data set can also be obtained from the aggregate indicator data table based on the object identifier corresponding to the target data.

[0081] Step 104: Determine the validity of the target data based on the waterline data and the consumed data set.

[0082] In the embodiment of the present disclosure, after obtaining the waterline data and the consumed data set, the validity of the target data can be determined based on the waterline data and the consumed data set. Exemplarily, when the storage time of the target data is greater than the waterline data and is not in the consumed data set, it indicates that the target data is new data and has not participated in the calculation of the aggregation index, and the target data can be considered to be valid data.

[0083] Step 106, when the target data is valid, perform a calculation task based on the aggregated index value and the target data, and update the aggregated index value in the aggregated index data table according to the calculation result.

[0084] In the disclosed embodiments, when the target data is valid, calculation tasks can be performed based on the target data and the aggregation index value, for example: adding newly generated data or subtracting invalid data to obtain calculation results, and the calculation results are updated in the aggregation index data table as the new aggregation index value of the target object.

[0085] The above data processing method, when the computing task corresponding to the statistical time window obtains the target data, obtains the aggregated data corresponding to the object identifier from the aggregated index data table based on the object identifier corresponding to the target data, and the aggregated data includes the waterline data corresponding to the computing task, the aggregated index value, and the consumed data set corresponding to the computing task. The validity of the target data is determined according to the waterline data and the consumed data set, and when the target data is valid, the computing task is performed based on the aggregated index value and the target data, and the aggregated index value in the aggregated index data table is updated according to the calculation result. The data processing method provided by the embodiment of the present disclosure stores various aggregated index values ​​corresponding to each object through the aggregated index data table, and obtains new data in real time through the statistical time window to update the aggregated index value in real time, that is, the aggregated index value is updated based on real-time data, so the amount of calculated data is small, which greatly improves the statistical efficiency and real-time performance of the aggregated index, and in the updating process, the validity of the acquired data is determined through the waterline data and the consumed data set, and the aggregated index value is updated only based on the valid data, so the interference of duplicate data and data not within the statistical period can be avoided, which can greatly improve the accuracy of the aggregated index value.

[0086] In an exemplary embodiment, the computing tasks corresponding to the statistical time window include accumulation tasks and subtraction tasks. The waterline data corresponding to the accumulation task is the storage time of the right boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the accumulation task is the set of right boundary data of the statistical time window that has been consumed; the waterline data corresponding to the subtraction task is the storage time of the left boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the subtraction task is the set of left boundary data of the statistical time window that has been consumed.

[0087] In the embodiment of the present disclosure, the computing tasks corresponding to the statistical time window include an accumulation task for accumulating newly generated data and a subtraction task for subtracting invalid data. In the aggregation index data table, the same aggregation index value corresponds to the waterline data and the consumed data set corresponding to the two computing tasks.

[0088] Among them, the waterline data corresponding to the accumulation task is the storage time of the left boundary data of the statistical time window calculated most recently, and the left boundary data of the statistical time window is the data scanned at the left boundary. The waterline data corresponding to the subtraction task is the storage time of the right boundary data of the statistical time window calculated most recently, and the right boundary data of the statistical time window is the data scanned at the right boundary.

[0089] The consumed data set corresponding to the accumulation task includes the ID information of each data that has participated in the accumulation calculation of the target aggregation index, and the consumed data set corresponding to the subtraction task includes the ID information of each data that has participated in the subtraction calculation of the target aggregation index.

[0090] Exemplarily, the aggregation indicator data table can refer to the following Table 1. The aggregation indicator data table is used to store the aggregation indicator data of the object, including two columns: object identifier (uid) and aggregation indicator (including aggregation indicator key value (metric_key) and aggregation indicator value (metric_value) fields). The object identifier can be set as the primary key. In order to ensure the accuracy of the calculation results, that is, the data passing through the left and right window boundaries is calculated only once, the embodiment of the present disclosure sets a water level field and a consumed data set field for the aggregation indicator of each object in the aggregation indicator data table, respectively identifying the most recently calculated left and right boundary data.

[0091] In Table 1, the water level field corresponding to the subtraction task is low_water_mark (for clear description, referred to as lwm in the following embodiment), the water level field corresponding to the accumulation task is high_water_mark (for clear description, referred to as hwm in the following embodiment), the consumed data set field corresponding to the subtraction task is low_water_mark_ids (for clear description, referred to as lids in the following embodiment), and the consumed data set field corresponding to the accumulation task is high_water_mark_ids (for clear description, referred to as hids in the following embodiment).

[0092] Table 1

[0093]

[0094] In the disclosed embodiment, the calculation status of the aggregated indicator can be marked by two watermark data, lwm and hwm, which come from the storage time CT of the basic data table. The lwm and hwm of the aggregated indicator are logical times in the business, and the time span between them can be greater than the length of the statistical time window in actual operation. This design can ensure that the aggregated indicator data of the complete cycle can be counted.

[0095] In order to ensure that the same object performs operations in the same synchronization, two calculation flow jobs are used when executing the calculation, which are responsible for the addition and subtraction operations of the aggregated indicator values ​​respectively. When writing data, the distributed pessimistic transaction characteristics of the distributed database are used. The entire calculation process can ensure the transactionality of the operation and the reentrancy of the calculation.

[0096] In an exemplary embodiment, the calculation task is an accumulation task. As shown in FIG. 5 , in step 104, determining the validity of the target data according to the waterline data and the consumed data set may include the following steps 502 to 504, wherein:

[0097] Step 502, obtaining the storage time and storage identification of the target data;

[0098] Step 504: If the storage time is greater than the waterline data corresponding to the accumulation task, and the storage identifier is not recorded in the consumed data set corresponding to the accumulation task, it is determined that the target data is valid.

[0099] In the disclosed embodiment, the storage time and storage identification (data ID information) of the target data can be obtained from the basic database, and whether the target data is valid can be determined based on the waterline data and the consumed data set corresponding to the cumulative task, as well as the storage time and storage identification of the target data.

[0100] Exemplarily, the data read by the accumulation task pointer responsible for scanning the right boundary of the statistical time window is the latest, such as the data set ids at the storage time CT1 (there may be multiple rows at the same time, ids is the set of data storage identifiers, and id is the data identifier). It determines whether the read data meets the accumulation conditions and performs subsequent operations. It should be noted that when an aggregate indicator is first written to the aggregate indicator data table, the storage time is set to its lwm and hwm, and the ids are set to lids and hids. If it is not the first write, the data storage time is required to be greater than hwm and the ids are not in hids in order to participate in the accumulation calculation. This ensures that the calculation of data with the same storage time is reentrant and will only be calculated once, and there will be no problem of repeated addition.

[0101] In an exemplary embodiment, the calculation task is a subtraction task. As shown in FIG. 6 , in step 104, the validity of the target data is determined according to the waterline data and the consumed data set, including the following steps 602 to 604, wherein:

[0102] Step 602, obtaining the storage time and storage identification of the target data;

[0103] Step 604: If the warehousing time is greater than the difference between the current calculation time and the window length of the time window, and the warehousing time is greater than the water level data corresponding to the subtraction task, and the warehousing identifier is not recorded in the consumed data set corresponding to the subtraction task, then the target data is determined to be valid.

[0104] In the disclosed embodiment, the entry time and entry identification (data ID information) of the target data can be obtained from the basic database, and whether the target data is valid can be determined based on subtracting the waterline data and the consumed data set corresponding to the task, as well as the entry time and entry identification of the target data.

[0105] In an exemplary embodiment, the data read by the calculation task pointer responsible for scanning the left boundary of the statistical time window is expired, such as the data set ids at the storage time CT2 (there may be multiple rows at the same time), and it determines whether the read data meets the subtraction conditions and performs subsequent operations. The orderliness of the original data and the design of the fixed sliding window ensure that the data passing the left boundary must have passed the right boundary, that is, the accumulation operation must have been implemented in advance, so there will be no situation where the aggregation index data table does not have the aggregation index value of the object. In order to avoid repeated subtraction, the embodiment of the present disclosure requires that the CT2 time of expired data must not only be less than (current calculation time-window length N) but also greater than lwm and ids are not in lids before performing the subtraction operation, otherwise the calculation operation after this scan will be ignored, indicating that the relevant data has been consumed and does not need to be repeated.

[0106] The disclosed embodiment provides a statistical method for aggregate indicators based on a distributed database. The aggregate indicator values ​​of all objects can be calculated offline in advance at one time when the data is initialized, and stored in a distributed relational database. A sliding statistical time window is defined for the constantly changing statistical period. New data passing the right boundary of the statistical time window is directly accumulated and updated, and expired data passing the left boundary is subtracted and updated. The aggregate indicator value of an aggregate indicator is dynamically maintained, and external access services are directly provided on-site, which is simple and efficient. Specifically, the incremental update mainly includes addition and subtraction operations: 1) If there is new data in the object within the statistical time window, an accumulation transaction is directly performed on the current aggregate indicator value; 2) If the object has expired data within the statistical time window, a subtraction transaction is directly performed on the current aggregate indicator value.

[0107] The above addition and subtraction operations are two independent operations and will be executed in two independent transactions, which can ensure that the changes in the aggregate indicator value always remain atomic and consistent. The computing task continuously monitors the latest data and continuously maintains the latest aggregate indicator value based on the real-time advancement of the change amount and the movement of the statistical time window, so that the latest cumulative value can be obtained in real time.

[0108] In an exemplary embodiment, in step 106, after the aggregation index value in the aggregation index data table is updated according to the calculation result, the method further includes the following steps:

[0109] According to the storage time corresponding to the target data, the waterline data corresponding to the computing task included in the aggregated data in the aggregated indicator data is updated; the storage identifier corresponding to the target data is added to the consumed data set corresponding to the computing task included in the aggregated data in the aggregated indicator data.

[0110] In the embodiment of the present disclosure, after the aggregation index value is updated based on the target data, the storage time of the target data can be used as the waterline data corresponding to the calculation task, and the waterline data corresponding to the aggregation index value in the aggregation index data table is updated. Exemplarily, when the storage time of the target data is CT1, after the aggregation index value is accumulated based on the target data, the storage time of the target data can be updated to the hwm corresponding to the aggregation index value; or, after the aggregation index value is subtracted based on the target data, the storage time of the target data can be updated to the lwm corresponding to the aggregation index value.

[0111] At the same time, the storage identification corresponding to the target data can be added to the consumed data set corresponding to the aggregate index value. Exemplarily, when the storage identification of the target data is id1, after the aggregate index value is accumulated based on the target data, the storage identification id1 of the target data can be added to the hids corresponding to the aggregate index value; or, after the aggregate index value is subtracted based on the target data, the storage identification id1 of the target data can be added to the lids corresponding to the aggregate index value.

[0112] In order to enable those skilled in the art to better understand the embodiments of the present disclosure, the embodiments of the present disclosure are described below through specific examples. In this example, a distributed fusion database HTAP (Hybrid Transactional and Analytical Processing) is introduced as a distributed database to undertake the basic data collected in real time, realize the storage and calculation of massive data, and finally provide high-performance data indicator access services. The distributed fusion database supports distributed transactions, and can also ensure efficient read and write performance in high-concurrency scenarios. At the same time, it has excellent scalability, supports massive storage without losing access performance, and can cope with continuously growing business needs.

[0113] Basic data tables and aggregated indicator data tables are pre-established in the distributed fusion database. The basic data table stores the self-incrementing row number ID, data record number (input identification), user ID, and input time CT value; the aggregated indicator data table stores the user ID, the window size of the statistical time window, the aggregated indicator value, the lwm corresponding to the subtraction task, the hwm corresponding to the accumulation task, the consumed data set corresponding to the subtraction task, and the consumed data set corresponding to the accumulation task.

[0114] 7 and 8 , the implementation process of this example is realized through a detail collection module, an indicator calculation module and a real-time data service module.

[0115] Taking the total amount of consumption as an example, the detailed collection module can collect the user's consumption behavior data from the online consumption log in real time, and store it in the basic data table pre-established in the relational distributed database. This can solve the problems of data disorder and duplication at the same time and complete the generation of basic data.

[0116] In the indicator calculation module, reentrant calculation and persistent storage of aggregate indicators are supported. At the beginning, based on historical data, the basic indicators are calculated once using offline statistics, imported into the distributed database, and then calculated based on this result. After that, the two calculation tasks responsible for accumulation and subtraction read the left and right boundary data of the statistical time window respectively and perform operations. The database cluster is set to pessimistic lock transaction mode in advance. When the calculation job is written to the database, the INSERT ONDUPLICATE KEY UPDATE statement is executed to complete the addition and subtraction operations of the aggregate indicator value, which can ensure the atomicity and isolation of the operation transaction.

[0117] As shown in Figure 9, it is a schematic diagram of the calculation process of the indicator calculation module. The current aggregation value is 1000 (window length N=T2-T1). As the statistical time window slides forward, 200 new data enter the statistical time window at T4 and 100 old data expire at T3. The two independent calculation operations of accumulation and subtraction are independent transactions and do not affect each other. Therefore, the result value is first accumulating 200 and then subtracting 100, and first subtracting 100 and then accumulating 200. The aggregation results of the two are the same, 1100 (window N=T6-T5).

[0118] In the real-time data service module, the database write operation does not block the read operation. During the calculation, the data is also served to the outside in real time without affecting the availability of online services. Point query of primary key users has high enough concurrent access performance.

[0119] The disclosed embodiment aims to provide a general, efficient and concise method for calculating the aggregated indicators of users within a period of time when conducting relevant analysis such as user portraits, marketing activities, and report dashboards in cloud service products or the Internet. Compared with the prior art, the advantages and effects are as follows: 1) Supporting the rapid calculation of aggregated indicators within the statistical time and obtaining the latest aggregated data in real time. 2) The calculation of aggregated indicators and the access to business are separated and decoupled, effectively reducing the complexity of business programs. 3) In addition to the first basic data statistics, each subsequent update of the database side only needs to complete a relatively light incremental calculation, and each statistics will not be completely completed by relying on the full calculation of the database, which will cause bottlenecks in database IO, CPU and other resources, effectively lowering the application threshold. 4) By introducing the HTAP distributed database as the storage base, it not only supports the storage of massive data, but also supports transactions and reentrant calculations. When encountering network interruptions, database jitters, or calculation flow failures and restarts, the data will not be lost or recalculated, which can ensure the security of the data and the stability of the service. 5) There is no need to introduce a cache database, and the result indicators are directly stored in the same database. High-concurrency access to the business can be achieved based on the primary key point query access of the table, ensuring low-latency access and stability. The indicator data is instantly persisted, and the data will not be lost in the event of failures such as downtime or power outages. 6) Support for big data storage can support the large-scale development of the business without adjusting the storage architecture. 7) There is no requirement to introduce a real-time computing framework, and the usage and management thresholds are relatively low, which is easy to promote. But in fact, it also supports the introduction of a real-time computing framework to optimize the computing module and further improve the computing efficiency of the embodiments of the present disclosure, that is, the embodiments of the present disclosure have flexible scalability.

[0120] In short, the problem of accurately and efficiently aggregating indicators in real time in massive data scenarios and providing OLTP business access immediately on the spot can be solved through the embodiments of the present disclosure. The efficiency of generating aggregated values ​​over a long time span can be improved to meet the growing real-time operation requirements. For example: In order to quickly promote payment-related marketing activities, a certain billing product needs to provide real-time total consumption amount indicators for each user in the past 5 years. By using the data processing method provided by the embodiments of the present disclosure, the indicator value can be counted in real time, and the online service can query the value in real time. As time goes by, this value can be updated in real time, which can be used as a basis for real-time evaluation of VIP users and for targeted marketing activities.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0123] In an exemplary embodiment, as shown in FIG10 , a data processing device is provided, including: an acquisition module 1002, a determination module 1004 and a processing module 1006, wherein:

[0124] The acquisition module 1002 is used to acquire the aggregated data corresponding to the object identifier from the aggregated indicator data table based on the object identifier corresponding to the target data when the computing task corresponding to the statistical time window acquires the target data, wherein the aggregated data includes the waterline data corresponding to the computing task, the aggregated indicator value, and the consumed data set corresponding to the computing task;

[0125] A determination module 1004, configured to determine the validity of the target data according to the waterline data and the consumed data set;

[0126] The processing module 1006 is used to perform the calculation task based on the aggregation index value and the target data when the target data is valid, and update the aggregation index value in the aggregation index data table according to the calculation result.

[0127] The data processing device provided by the embodiment of the present disclosure stores various types of aggregation index values ​​corresponding to each object through an aggregation index data table, and obtains new data through real-time scanning of a statistical time window to update the aggregation index value in real time, that is, the aggregation index value is updated based on real-time data, so the amount of calculated data is small, which greatly improves the statistical efficiency and real-time performance of the aggregation index, and during the update process, the validity of the acquired data is determined through waterline data and consumed data sets, and the aggregation index value is updated only based on valid data, thereby avoiding interference from duplicate data and data that is not within the statistical period, and can greatly improve the accuracy of the aggregation index value.

[0128] In one embodiment, the computing tasks corresponding to the statistical time window include accumulation tasks and subtraction tasks. The waterline data corresponding to the accumulation task is the storage time of the right boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the accumulation task is the set of right boundary data of the statistical time window that has been consumed; the waterline data corresponding to the subtraction task is the storage time of the left boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the subtraction task is the set of left boundary data of the statistical time window that has been consumed.

[0129] In one embodiment, the calculation task is the accumulation task, and the determination module 1004 is specifically used to: obtain the warehousing time and warehousing identifier of the target data; if the warehousing time is greater than the waterline data corresponding to the accumulation task, and the warehousing identifier is not recorded in the consumed data set corresponding to the accumulation task, then it is determined that the target data is valid.

[0130] In one embodiment, the calculation task is the subtraction task, and the determination module 1004 is specifically used to: obtain the warehousing time and warehousing identifier of the target data; if the warehousing time is greater than the difference between the current calculation time and the window length of the time window, and the warehousing time is greater than the water level data corresponding to the subtraction task, and the warehousing identifier is not recorded in the consumed data set corresponding to the subtraction task, then it is determined that the target data is valid.

[0131] In one of the embodiments, after the aggregation index value in the aggregation index data table is updated according to the calculation result, the device further includes an updating module for:

[0132] According to the storage time corresponding to the target data, the waterline data corresponding to the computing task included in the aggregated data in the aggregated indicator data is updated; the storage identifier corresponding to the target data is added to the consumed data set corresponding to the computing task included in the aggregated data in the aggregated indicator data.

[0133] In one embodiment, the statistical time window has a left boundary and a right boundary, the left boundary corresponds to a subtraction task, and the right boundary corresponds to an accumulation task, and the device further includes a scanning module for:

[0134] Slide the statistical time window according to a preset time step; scan data in a basic data table based on the sliding of the left boundary and / or the right boundary of the statistical time window, and use the scanned data as target data, wherein the basic data table includes a storage time field, and the storage time corresponding to the target data is within the sliding range of the left boundary and / or the right boundary.

[0135] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0136] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0137] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0140] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0144] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: When the computing task corresponding to the statistical time window obtains the target data, based on the object identifier corresponding to the target data, the aggregated data corresponding to the object identifier is obtained from the aggregated indicator data table, the aggregated data including the watermark data corresponding to the computing task, the aggregated indicator value, and the consumed data set corresponding to the computing task; Determining the validity of the target data according to the waterline data and the consumed data set; When the target data is valid, the calculation task is performed based on the aggregation index value and the target data, and the aggregation index value in the aggregation index data table is updated according to the calculation result.

2. The method according to claim 1, characterized in that: The calculation tasks corresponding to the statistical time window include accumulation tasks and subtraction tasks. The waterline data corresponding to the accumulation task is the storage time of the right boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the accumulation task is the set of right boundary data of the statistical time window that has been consumed; The waterline data corresponding to the subtraction task is the storage time of the left boundary data of the statistical time window calculated most recently, and the consumed data set corresponding to the subtraction task is the set of the left boundary data of the statistical time window that has been consumed.

3. The method according to claim 2, characterized in that The calculation task is the accumulation task, and determining the validity of the target data according to the waterline data and the consumed data set includes: Obtaining the storage time and storage identification of the target data; If the storage time is greater than the waterline data corresponding to the accumulation task, and the storage identifier is not recorded in the consumed data set corresponding to the accumulation task, it is determined that the target data is valid.

4. The method according to claim 2, characterized in that: The calculation task is the subtraction task, and determining the validity of the target data according to the waterline data and the consumed data set includes: Obtaining the storage time and storage identification of the target data; If the warehousing time is greater than the difference between the current calculation time and the window length of the time window, and the warehousing time is greater than the waterline data corresponding to the subtraction task, and the warehousing identifier is not recorded in the consumed data set corresponding to the subtraction task, then the target data is determined to be valid.

5. The method according to claim 3 or 4, characterized in that: After the aggregation index value in the aggregation index data table is updated according to the calculation result, the method further includes: According to the storage time corresponding to the target data, update the waterline data corresponding to the computing task included in the aggregated data in the aggregated indicator data; The storage identification corresponding to the target data is added to the consumed data set corresponding to the computing task included in the aggregated data in the aggregated indicator data.

6. The method according to claim 1, characterized in that The statistical time window has a left boundary and a right boundary, the left boundary corresponds to a subtraction task, and the right boundary corresponds to an accumulation task, and the method further includes: Slide the statistical time window according to a preset time step; Based on the sliding of the left boundary and / or the right boundary of the statistical time window, data is scanned in the basic data table, and the scanned data is used as the target data. The basic data table includes an entry time field, and the entry time corresponding to the target data is within the sliding range of the left boundary and / or the right boundary.

7. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire, when the computing task corresponding to the statistical time window acquires the target data, aggregate data corresponding to the object identifier from the aggregate indicator data table based on the object identifier corresponding to the target data, wherein the aggregate data includes waterline data corresponding to the computing task, an aggregate indicator value, and a consumed data set corresponding to the computing task; A determination module, configured to determine the validity of the target data according to the waterline data and the consumed data set; A processing module is used to perform the calculation task based on the aggregation index value and the target data when the target data is valid, and to update the aggregation index value in the aggregation index data table according to the calculation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images