A service data processing method, device and equipment

Abstract

The embodiment of the specification discloses a kind of service data processing method, device and equipment.Solution includes: determining the first local result corresponding to the basic time interval from the starting time, the first local result is obtained by the overall processing to the service data set in the basic time interval;Determine a plurality of different quantity values, each quantity value respectively indicates the corresponding number of unit time after the basic time interval;Respectively for each quantity value, and parallelly execute: obtain the service data set in the corresponding number of unit time indicated by the quantity value, the overall processing is carried out to the service data set, the second local result corresponding to the corresponding number of unit time is obtained, according to the first local result and the second local result, the overall processing is carried out, and the global result of the end time corresponding to the quantity value is obtained.

Description

Technical Field

[0001] This specification relates to the field of big data technology, and in particular to a business data processing method, apparatus, and device. Background Technology

[0002] With the rapid development of Internet technology, various applications have emerged, bringing great convenience to users. As the user base and business areas expand, the accumulation of business data is also continuous and large-scale, with a large amount of new business data added every day, which is especially evident for large and medium-sized applications.

[0003] To better serve users, these application service providers will conduct business data analysis more frequently. A typical scenario is to calculate the latest full window data for each day from a specified starting point in history to the current day. This full window data is obtained by processing the original business data as a whole, such as data deduplication and data matching.

[0004] Currently, the aforementioned full window data is recalculated for each day based on all business data within the time range from the start time to that day. This consumes a lot of computing resources and is inefficient.

[0005] Therefore, for the above scenario, a more cost-effective and efficient business data processing solution is needed. Summary of the Invention

[0006] This specification provides one or more embodiments of a business data processing method, apparatus, device, and storage medium to solve the following technical problem: the need for a lower cost and higher efficiency business data processing solution.

[0007] To solve the above-mentioned technical problems, one or more embodiments of this specification are implemented as follows:

[0008] This specification provides a business data processing method according to one or more embodiments, including:

[0009] Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0010] Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval;

[0011] For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; and performing the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the stated quantity value.

[0012] This specification provides a business data processing apparatus according to one or more embodiments, comprising:

[0013] The first partial result determination module determines the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0014] The different quantity value determination module determines multiple different quantity values, each of which indicates a corresponding number of unit times following the basic time interval;

[0015] The parallel global result determination module performs the following operations in parallel for each of the quantity values: obtaining the set of business data within a unit time period corresponding to the quantity value; performing the overall processing on the set of business data to obtain the second local result corresponding to the unit time period; and performing the overall processing based on the first local result and the second local result to obtain the global result at the end time corresponding to the quantity value.

[0016] This specification provides one or more embodiments of a business data processing device, comprising:

[0017] At least one processor; and,

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0020] Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0021] Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval;

[0022] For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; and performing the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the stated quantity value.

[0023] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0024] Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0025] Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval;

[0026] For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; and performing the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the stated quantity value.

[0027] The above-described at least one technical solution adopted in one or more embodiments of this specification can achieve the following beneficial effects: it can effectively reuse the business data processing results within the basic time interval, thus eliminating the need to reprocess the entire dataset every day, improving efficiency, saving processing resources, and reducing costs. Furthermore, it provides a business data processing flow template (e.g., represented by an SQL statement template) for reusing the basic time interval and appending dynamic length subsequent time intervals. Based on multiple different quantity values, by instantiating the processing flow template (e.g., represented by corresponding SQL statements), it can flexibly and efficiently calculate the global results (e.g., full window data) corresponding to multiple days in parallel, avoiding the need to serially follow up the latest global results day by day. It can also use these quantity values ​​to control changes in the basic time interval so as to reuse updated global results in a timely manner, further improving efficiency. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a business data processing method provided in one or more embodiments of this specification;

[0030] Figure 2 This is a schematic diagram illustrating the principle of a scheme for reusing historical business data processing results and serially tracing back the latest global results day by day, as provided in one or more embodiments of this specification.

[0031] Figure 3 A schematic diagram illustrating the principle of a scheme for reusing historical business data processing results based on quantitative variables and dynamic basic time intervals, provided for one or more embodiments of this specification;

[0032] Figure 4 In a practical application scenario provided by one or more embodiments of this specification, for Figure 1 A schematic diagram illustrating the principle of the specific implementation scheme of the method;

[0033] Figure 5 A schematic diagram of the structure of a business data processing apparatus provided for one or more embodiments of this specification;

[0034] Figure 6 This is a schematic diagram of the structure of a business data processing device provided for one or more embodiments of this specification. Detailed Implementation

[0035] This specification provides a business data processing method, apparatus, device, and storage medium through its embodiments.

[0036] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0037] As mentioned in the background section, recalculating the aforementioned full window data (also known as the global result up to the current day, or simply the global result) for each day based on all business data within the time range from the start time to the current day consumes significant computing resources and is inefficient. To better understand why this requirement and problem exist, an example is provided for explanation.

[0038] Suppose an activity or feature has been online for a period of time (e.g., one month) and requires report analysis to statistically analyze the cumulative user usage of the activity or feature from the launch date to each day thereafter. Even if the same user uses the activity or feature multiple times, only one instance is counted, and all other instances are considered duplicate data. Therefore, this strategy requires corresponding deduplication of global business data. The current processing method might, for example, first obtain all business data for the first day after launch, then deduplicate it to obtain the global result up to day one; then obtain all business data for the first and second days after launch, then deduplicate it again to obtain the global result up to day two; then obtain all business data for the first, second, and third days after launch, then deduplicate it again to obtain the global result up to day three; and so on, recalculating the global result up to that day based on the corresponding full business data for each day. While logically simple and easy to understand, this approach consumes significant computing resources and is inefficient.

[0039] To address this issue and reduce resource consumption, the applicant initially considered reusing the global results up to the previous day, and then using these global results to calculate the global results up to the next day. Since the reused global results are already deduplicated, adding the business data from the next day to the global results for another deduplication reduces resource consumption. However, the applicant found that this approach can only be performed serially, tracing back to the next day's business data day by day to obtain the updated global results, thus remaining inefficient. Therefore, the applicant further improved the solution by constructing a business data processing template that reuses historical global results and appends dynamically long subsequent time intervals. This template is implemented as an example of an SQL statement template specifying the quantity variable for this dynamic length, enabling flexible parallel calculation of global results for multiple days, balancing efficiency and low cost.

[0040] Based on this overall approach, the following section will further explain the scheme of this application.

[0041] Figure 1This is a flowchart illustrating a business data processing method provided in one or more embodiments of this specification. All or at least most of the actions included in this process can be executed in a business database or corresponding computing engine / automation server, for example, in the form of SQL statements or automation scripts, which helps to improve efficiency. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0042] Figure 1 The process includes the following steps:

[0043] S102: Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0044] In one or more embodiments of this specification, the start time can be open (e.g., tracing back along the timeline) or explicitly specified, such as starting from a specific historical day (e.g., the day the target activity or feature went live). The start time is the beginning of that day. The first partial result is the global result up to the end of the base time interval, and is also the global result to be reused subsequently. The first partial result can be... Figure 1 If the process is pre-calculated before execution, it can be called multiple times as needed without repeated calculations, thus improving efficiency.

[0045] In one or more embodiments of this specification, the length of a unit of time is specified, for example, one unit of time is one day. The unit of time can also be defined as other time lengths as needed. Figure 1 In the process, the length of a unit of time is fixed, which makes it easy to measure the length of other time intervals using the unit of time as a standard.

[0046] In this scenario, the initial base time interval can be controlled to a length of one unit of time, thus consisting of the first unit of time starting from the initial moment. Of course, if a global result corresponding to a longer time interval has been obtained, the base time interval can also be controlled to that longer time interval to directly reuse the global result, meaning the base time interval can also consist of multiple units of time. Furthermore, following this approach, as the process progresses or iterates, the length of the base time interval can be dynamically adjusted (e.g., dynamically adjusted based on updated global results). However, in… Figure 1 In the single-round execution of the process, the length of the basic time interval can be fixed to facilitate overall control, which may help improve overall efficiency.

[0047] In one or more embodiments of this specification, "overall processing" refers to processing that requires integrating all data from one or more datasets. If new data is added to this dataset, the previously obtained overall processing results may become inaccurate, potentially necessitating further overall processing on the new dataset. For example, in the data deduplication process mentioned above, newly added data may duplicate previously deduplicated data. Similarly, overall processing can also include other processes such as data matching and model training.

[0048] S104: Determine multiple different quantity values, each of which indicates a corresponding number of unit times following the base time interval.

[0049] In one or more embodiments of this specification, multiple subsequent time intervals of different lengths are divided immediately following the base time interval by multiple different quantity values, thereby realizing a dynamic length of subsequent time intervals. These subsequent time intervals of different lengths can be decoupled and used in conjunction with the base time interval, reusing the global results corresponding to the base time interval, and calculating the global results corresponding to multiple longer time intervals of different lengths in parallel. Thus, it is not necessary to calculate and update the global results serially on a unit time basis (e.g., on a day-by-day basis).

[0050] Following a similar approach, multiple numerical values ​​can be used in conjunction with base time intervals of varying lengths for parallel computation, executing in multiple rounds. Figure 1 When following the process, you can consider doing the following.

[0051] In one or more embodiments of this specification, the quantity values ​​are positive integers to be used in conjunction with unit time, avoiding further division of a unit time. To more systematically advance the parallel processing of historical business data from far to near, multiple different quantity values ​​can be a relatively regular sequence of positive integers, such as consecutive positive integers or positive integers forming an arithmetic sequence. For example, assuming three different quantity values ​​are determined, namely positive integers 1, 2, and 3, then the first quantity value "1" indicates the first unit time after the basic time interval, the second quantity value "2" indicates the first and second unit times after the basic time interval, and the third quantity value "3" indicates the first, second, and third unit times after the basic time interval.

[0052] Furthermore, the base time interval itself can be determined by controlling the quantity variable, thus enabling more flexible concurrency control. Specifically, as needed, any unit time reflecting the end time can be specified (for example, specifying a later day), and the global result to be calculated up to the end time of that unit time can be freely controlled based on that unit time and the quantity variable, so as to reuse the latest obtained global result.

[0053] S106: For each of the stated quantity values, execute in parallel the following steps: obtain the set of business data within a unit time period corresponding to the quantity value; perform the overall processing on the business data set to obtain the second partial result corresponding to the corresponding quantity within a unit time period; and perform the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the quantity value. That is, obtain the global result up to the end time.

[0054] In one or more embodiments of this specification, a unified business data processing flow template is constructed so that global results can be calculated in parallel at multiple time points (with different end times corresponding to different quantity values ​​under the same base time interval) based on different quantity values.

[0055] This business data processing workflow template can include the main actions executed in parallel in step S106. If necessary, it can also include more actions, such as the action of determining the first local result in step S102. The template can include a set quantity variable. By controlling the change of this quantity variable, multiple different quantity values ​​can be obtained. That is, by assigning values ​​to this quantity variable multiple times, multiple instances corresponding to the template can be obtained. Step S106 represents the parallel execution of multiple such instances.

[0056] In one or more embodiments of this specification, the first partial result can be pre-calculated and therefore directly reused during parallel execution. The "set of business data per unit time corresponding to the quantity indicated by this quantity value" refers to the business data added on top of the base time interval. Since the first partial result has already been processed as a whole, the added business data itself is processed first to obtain the second partial result. This avoids prematurely mixing the data, which would reduce overall processing efficiency and consume more resources. Then, based on the first and second partial results, overall processing is performed again, improving efficiency. The end time corresponding to this quantity value is the end time of the last unit time indicated by this quantity value.

[0057] Taking the overall processing, which includes data deduplication, as an example, it is necessary to take the union of the first local result and the second local result, and then perform data deduplication to obtain the global result at the end time corresponding to the quantity value.

[0058] In one or more embodiments of this specification, the end time corresponding to the quantity value in step S106 can be specified. Then, based on the end time and the quantity value, the basic time interval is determined, thereby enabling the determination of the basic time interval through quantity variable control.

[0059] pass Figure 1 This method effectively reuses business data processing results within a basic time interval. This eliminates the need for daily reprocessing of the entire dataset, improving efficiency, saving processing resources, and reducing costs. Furthermore, it provides business data processing workflow templates for reusing the basic time interval and appending dynamically longer subsequent time intervals (e.g., represented by SQL statement templates or automated script templates). By instantiating these workflow templates based on multiple different quantitative values ​​(e.g., represented by corresponding SQL statements or automated scripts), it can flexibly and efficiently calculate the global results for multiple days (e.g., the entire window of data) in parallel. This avoids the need for sequentially tracing back to the latest global results day by day. These quantitative values ​​can also be used to control changes in the basic time interval to ensure timely reuse of updated global results, further improving efficiency.

[0060] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation schemes and extension schemes of this method, which will be further explained below.

[0061] Based on the preceding explanation, to make things more intuitive, the applicant's proposed serial and parallel execution schemes are compared and illustrated graphically. See [link to relevant documentation]. Figures 2-4 For example, one day is taken as one unit of time.

[0062] Figure 2 This is a schematic diagram illustrating the principle of a scheme for reusing historical business data processing results and serially tracing back to the latest global results day by day, as provided in one or more embodiments of this specification.

[0063] exist Figure 2The structure consists of five layers from top to bottom, described layer by layer. The first layer represents the input business data table, storing all the necessary business data. The second layer represents the set of business data within a basic time interval, yielding a reusable global result. The third layer appends one day's worth of business data, reusing this global result for overall processing to obtain an updated global result (referred to as the third-layer global result). The fourth layer further appends one day's worth of business data to the third layer, reusing the third-layer global result for overall processing to obtain the fourth-layer global result. The fifth layer further appends one day's worth of business data to the fourth layer, reusing the fourth-layer global result for overall processing to obtain the fifth-layer global result.

[0064] As you can see, Figure 2 The process is performed sequentially. It is necessary to wait until the global result appended the previous day is calculated before the next day's result can be appended in order to reuse the global result.

[0065] Figure 3 This is a schematic diagram illustrating the principle of a scheme for reusing historical business data processing results based on quantitative variables and dynamic basic time intervals, provided in one or more embodiments of this specification.

[0066] exist Figure 3 The system has three layers. In each layer, the dark horizontal bars represent the base time interval, and the light horizontal bars represent the total time interval. The total time interval is the same for all three layers, but it can be specified as needed (e.g., specifying the last day). As you can see, the values ​​of the quantity variables in each layer (i.e., different quantity values) are 3, 2, and 1 days, respectively, thus controlling the base time interval to have different lengths, increasing by 1 day with each layer. Therefore, by controlling based on quantity variables, global data up to the end of any given day can be reused to calculate the global data for the total time interval.

[0067] Figure 4 In a practical application scenario provided by one or more embodiments of this specification, for Figure 1 A schematic diagram illustrating the principle of the specific implementation plan of the method.

[0068] exist Figure 4 In the middle, similar reference Figure 3 Understanding, and Figure 3The main difference lies in the fact that each layer specifies a total time interval of different lengths to allow for parallel computation of global results up to different days (three consecutive days are selected in the example in the diagram). In the diagram, the values ​​of the quantity variables for each layer (i.e., different quantity values) are 1, 2, and 3 days, respectively. Combined with the specification of the total time interval, this ensures that the base time interval remains the same. Following this approach, the number of concurrent executions can be the maximum value of the quantity variable at present.

[0069] In one or more embodiments of this specification, the number of concurrent executions needs to be adapted to the computing power. Figure 1 The concurrency level can be determined by the number of different values ​​in the process. However, in practical applications, the number of units of time after the base time interval may be large (for example, if the total time range is one month, and each day is one unit of time, there are 30 units of time; similarly, the number of units of time may be even greater, requiring calculation of the global result at the end of each unit of time). It is difficult to obtain the global result for each target time (e.g., the end time of each unit of time starting from the start time) through a single round of concurrent calculation. Therefore, it can be divided into multiple rounds, with each round obtaining the global result for a portion of the target times, until the global result for all target times is obtained.

[0070] Therefore, by performing the above parallel execution process in one or more rounds, we can obtain the global result of the end time of each unit of time within the time range from the start time to the specified end time (for example, the end time of the latest unit of time that has been passed, such as when it ended yesterday).

[0071] Based on this approach, when determining multiple different quantity values, specifically, all quantity values ​​within a specified range of positive integers can be identified. After executing each quantity value in parallel, based on the parallel execution results, a suitable range of positive integers is specified, ensuring it includes all quantity values, and the same parallel execution process is performed on these values. This process is repeated multiple times until the final endpoint is reached. Of course, in this scenario, after each round of execution, some updated global results are determined. These global results can be reused in the next round. Therefore, the length of the base time interval can be dynamically adjusted (especially controlled by the aforementioned quantity variables) to allow the quantity values ​​to coordinate with base time intervals of different lengths for parallel execution.

[0072] In one or more embodiments of this specification, in order to facilitate implementation and automated execution, this application specifically implements the above-mentioned business process template as an SQL statement template constructed based on the above-mentioned quantity variables. Then, by assigning the quantity variables to some of the above-mentioned quantity values ​​in the SQL statement template, SQL statements corresponding to each quantity value are generated accordingly, so as to complete the above-mentioned parallel execution process by executing each SQL statement.

[0073] The process of constructing an SQL statement template specifically includes: using the quantity variable and the specified unit time reflecting the end time (for example, the day as the current target, to calculate the global result up to that day), indirectly representing the basic time interval (that is, realizing the determination of the basic time interval through the quantity variable), and the time range from the basic time interval to the end time; constructing an SQL statement template based on the specified unit time reflecting the end time, the indirectly represented basic time interval, and the time range.

[0074] More intuitively, in a practical application scenario, we can compare and illustrate with specific SQL statements and templates. In this scenario, the global result reflects the user participation in the activity after data deduplication within the time range from the start time to the corresponding end time. Assuming that the overall processing mainly includes data deduplication, the global result will be referred to as the global deduplication result in the following examples.

[0075] First, assuming the solution mentioned in the background technology is adopted, that is, for each day, all business data up to that day is deduplicated again. Then, the following SQL statement can be used as an example to achieve this:

[0076] select active_id,user

[0077] from table1

[0078] where dt <= ${yyyyMMdd}

[0079] group by active_id,user”

[0080] Where active_id represents the activity identifier, user represents the user identifier, table1 represents the data table that stores all the business data to be used, dt represents the time range, and yyyyMMdd represents the day that is the current target. The global deduplication result to be calculated up to the end of that day is referred to as the global deduplication result corresponding to yyyyMMdd.

[0081] As you can see, the actions performed by these SQL statements are as follows: query the business data set up to and including the day yyyyMMdd from table1, and then perform data deduplication processing (and grouping accordingly) on the business attribute dimensions of active_id and user using the group by operator (similarly, other operators with deduplication capabilities can also be used) to obtain the global deduplication result corresponding to yyyyMMdd.

[0082] Similarly, assuming the global deduplication result is reused, and the business data for the next day is sequentially traced forward day by day, the following SQL statement can be used as an example:

[0083]

[0084]

[0085] Here, mv represents the data table that stores the historical global deduplication results.

[0086] As can be seen, the actions performed by these SQL statements are as follows: Query the business data set for the day yyyyMMdd from table1, then use the group by operator to remove duplicates, obtaining the first part of the global deduplication result; and also query the global deduplication result corresponding to the day before yyyyMMdd (i.e., yyyyMMdd-1) from mv, as the second part of the global deduplication result; then use the union all operator to take the union of these two parts of the global deduplication result, and use the group by operator to remove duplicates from this union, obtaining the global deduplication result corresponding to yyyyMMdd.

[0087] Furthermore, to achieve parallel execution and dynamically control the base time interval and appended subsequent time intervals, the following SQL statement template is constructed based on data variables:

[0088]

[0089] Here, N is the aforementioned quantitative variable. yyyyMMdd-N is taken as the last day of the basic time interval, thus indirectly representing the basic time interval, which is the entire time interval from that last day back to a certain starting time. Therefore, the basic time interval is controlled by N and yyyyMMdd. "dt<=${yyyyMMdd}and dt>=${yyyyMMdd+1-N}" indirectly represents the time range from the end of the aforementioned basic time interval to the end time (the end time of the day yyyyMMdd).

[0090] As can be seen, the SQL statement template performs the following actions: dynamically controlling the base time interval and the time range based on N and yyyyMMdd; querying the business data set within the time range from table1, and then using the group by operator to remove duplicates, resulting in... Figure 1 The second partial result in the process; and, also, the global deduplication result corresponding to the basic time interval is retrieved from the MV query, resulting in... Figure 1 The first local result in the process; then, the union all operator is used to take the union of the second local result and the first local result, and the group by operator is used to remove duplicates from the union, resulting in the global deduplication result corresponding to yyyyMMdd.

[0091] As needed, multiple values ​​can be assigned to yyyyMMdd in the SQL statement template (for example, assigning them the date values ​​of three consecutive days), and multiple values ​​can be assigned to N accordingly (for example, assigning them the values ​​1, 2, and 3). This will result in multiple SQL statements corresponding to different days. By executing these statements in parallel, the global deduplication results corresponding to these different days can be calculated. Furthermore, the growth of the base time interval can be controlled to continue to calculate the global deduplication results corresponding to more subsequent days in parallel.

[0092] Therefore, based on the SQL statement template provided in this application, business data can be processed concurrently according to resource and time requirements, and the execution of concurrent processes can be flexibly controlled. Historical processing results can be reused flexibly and efficiently, which effectively improves processing efficiency and helps to reduce resource consumption.

[0093] Based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above methods, such as... Figure 5 , Figure 6 As shown. The apparatus and equipment are capable of performing the above methods and related alternatives accordingly.

[0094] Figure 5 This specification provides a schematic diagram of the structure of a business data processing apparatus according to one or more embodiments, the apparatus comprising:

[0095] The first partial result determination module 502 determines the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0096] The different quantity value determination module 504 determines multiple different quantity values, each of which indicates a corresponding number of unit times following the basic time interval.

[0097] The parallel global result determination module 506 performs the following in parallel for each of the quantity values: obtaining the set of business data per unit time corresponding to the quantity value; performing the overall processing on the set of business data to obtain the second local result corresponding to the corresponding quantity per unit time; and performing the overall processing based on the first local result and the second local result to obtain the global result at the end time corresponding to the quantity value.

[0098] Optionally, the basic time interval consists of one unit of time or multiple consecutive units of time.

[0099] Optionally, the different quantity value determination module 504 obtains multiple different quantity values ​​by controlling the changes of a set quantity variable, and the basic time interval is determined by the quantity variable control.

[0100] Also includes:

[0101] SQL statement template determination module 508 determines the SQL statement template constructed based on the quantity variable;

[0102] The parallel SQL statement generation module 510 assigns the quantity variables to the respective quantity values ​​in the SQL statement template, and generates corresponding SQL statements for each quantity value, so as to complete the parallel execution process by executing each SQL statement.

[0103] Optionally, it also includes:

[0104] SQL statement template construction module 512, before determining the SQL statement template constructed based on the quantity variable, indirectly represents the basic time interval and the time range from the basic time interval to the end time by using the quantity variable and a specified unit time reflecting the end time.

[0105] The SQL statement template is constructed based on the specified unit time reflecting the end time, and the indirectly represented basic time interval and time range.

[0106] Optionally, the different quantity value determination module 504 determines all quantity values ​​within a specified positive integer value range, wherein the quantity values ​​are integers;

[0107] The parallel global result determination module 506, after executing in parallel for each of the quantity values, redetermines the positive integer value range based on the execution result, and within the redetermined positive integer value range, determines all quantity values ​​and performs the parallel execution process on them as well.

[0108] Optionally, the first local result determination module 502 dynamically adjusts the length of the base time interval so that the quantity value can be matched with base time intervals of different lengths to perform the parallel execution process.

[0109] Optionally, the parallel global result determination module 506 performs the parallel execution process in one or more rounds to obtain the global result of the end time of each unit time within the time range from the start time to the specified end time.

[0110] Optionally, the overall processing includes data deduplication processing;

[0111] The parallel global result determination module 506 performs the data deduplication process after taking the union of the first local result and the second local result.

[0112] Optionally, the data deduplication process is performed by using a group by operator for a specified business attribute in an SQL statement for a specified business data table.

[0113] Optionally, the specified business attributes include activity identifier and / or user identifier;

[0114] The global results reflect the deduplicated user participation in the activity within the time range from the start time to the corresponding end time.

[0115] Figure 6 This specification provides a schematic diagram of the structure of a business data processing device according to one or more embodiments, the device comprising:

[0116] At least one processor; and,

[0117] A memory communicatively connected to the at least one processor; wherein,

[0118] The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0119] Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0120] Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval;

[0121] For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; and performing the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the stated quantity value.

[0122] Based on the same idea, one or more embodiments of this specification also provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0123] Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole.

[0124] Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval;

[0125] For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; and performing the overall processing based on the first partial result and the second partial result to obtain the global result at the end time corresponding to the stated quantity value.

[0126] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0127] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0128] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0129] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0130] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A business data processing method, comprising: Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole. Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval; For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; reusing the first partial result; and performing the overall processing on the second partial result incrementally to obtain the global result at the end time corresponding to the stated quantity value.

2. The method as described in claim 1, wherein the basic time interval consists of one unit time or multiple consecutive unit times.

3. The method as described in claim 1, wherein determining a plurality of different quantity values ​​specifically includes: By controlling the changes of a set quantity variable, multiple different quantity values ​​can be obtained, and the basic time interval is determined by controlling the quantity variable.

4. The method of claim 3, further comprising: Determine the SQL statement template constructed based on the quantity variable; By assigning the quantity variable to each of the quantity values ​​in the SQL statement template, and generating SQL statements corresponding to each of the quantity values, the parallel execution process can be completed by executing each of the SQL statements.

5. The method of claim 4, wherein before determining the SQL statement template constructed based on the quantity variable, the method further comprises: The quantitative variables and the specified unit time reflecting the end time are used to indirectly represent the basic time interval and the time range from the basic time interval to the end time. The SQL statement template is constructed based on the specified unit time reflecting the end time, and the indirectly represented basic time interval and time range.

6. The method of claim 1, wherein determining a plurality of different quantity values ​​specifically includes: Determine all quantity values ​​within a specified range of positive integer values; After executing the steps in parallel for each of the stated quantity values, the method further includes: Based on the execution result, the range of positive integer values ​​is redefined. Within the redefined range of positive integer values, all numerical values ​​are determined and subjected to the parallel execution process.

7. The method of claim 1, further comprising: The length of the base time interval is dynamically adjusted so that the quantity value is matched with base time intervals of different lengths to perform the parallel execution process.

8. The method according to any one of claims 1 to 7, further comprising: By performing the parallel execution process in one or more rounds, the global result of the end time of each unit of time within the time range from the start time to the specified end time is obtained.

9. The method according to any one of claims 1 to 7, wherein the overall processing includes data deduplication processing; The overall processing based on the first partial result and the second partial result specifically includes: After taking the union of the first local result and the second local result, the data deduplication process is performed.

10. The method as described in claim 9, wherein the data deduplication process is performed by using a group by operator for a specified business attribute in an SQL statement for a specified business data table.

11. The method of claim 10, wherein the specified business attribute includes an activity identifier and / or a user identifier; The global results reflect the deduplicated user participation in the activity within the time range from the start time to the corresponding end time.

12. A business data processing apparatus, comprising: The first partial result determination module determines the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole. The different quantity value determination module determines multiple different quantity values, each of which indicates a corresponding number of unit times following the basic time interval; The parallel global result determination module performs the following in parallel for each of the quantity values: obtaining the business data set within a unit time period corresponding to the quantity value; performing the overall processing on the business data set to obtain the second local result corresponding to the corresponding quantity within a unit time period; reusing the first local result; and performing the overall processing on the second local result incrementally to obtain the global result at the end time corresponding to the quantity value.

13. The apparatus of claim 12, wherein the basic time interval comprises one unit time or multiple consecutive unit times.

14. The apparatus of claim 12, wherein the different quantity value determination module obtains multiple different quantity values ​​by controlling the change of a set quantity variable, and the basic time interval is determined by controlling the quantity variable.

15. The apparatus of claim 14, further comprising: The SQL statement template determination module determines the SQL statement template constructed based on the quantity variable. The parallel SQL statement generation module assigns the quantity variables to the respective quantity values ​​in the SQL statement template, and generates corresponding SQL statements for each quantity value, so as to complete the parallel execution process by executing each SQL statement.

16. The apparatus of claim 15, further comprising: The SQL statement template construction module, before determining the SQL statement template constructed based on the quantity variable, indirectly represents the basic time interval and the time range from the basic time interval to the end time by using the quantity variable and a specified unit time reflecting the end time. The SQL statement template is constructed based on the specified unit time reflecting the end time, and the indirectly represented basic time interval and time range.

17. The apparatus of claim 12, wherein the different quantity value determining module determines all quantity values ​​within a specified positive integer value range, wherein the quantity values ​​are integers; The parallel global result determination module, after executing in parallel for each of the quantity values, redetermines the positive integer value range based on the execution result, and within the redetermined positive integer value range, determines all quantity values ​​and performs the parallel execution process on them as well.

18. The apparatus of claim 12, wherein the first partial result determination module dynamically adjusts the length of the base time interval so that the quantity value is matched with base time intervals of different lengths to perform the parallel execution process.

19. The apparatus according to any one of claims 12 to 18, wherein the parallel global result determination module performs the parallel execution process in one or more rounds to obtain the global result of the end time of each unit time within the time range from the start time to the specified end time.

20. The apparatus according to any one of claims 12 to 18, wherein the overall processing includes data deduplication processing; The parallel global result determination module performs the data deduplication process after taking the union of the first local result and the second local result.

21. The apparatus of claim 20, wherein the data deduplication process is performed by using a group by operator for a specified business attribute in an SQL statement for a specified business data table.

22. The apparatus of claim 21, wherein the specified service attribute includes an activity identifier and / or a user identifier; The global results reflect the deduplicated user participation in the activity within the time range from the start time to the corresponding end time.

23. A business data processing device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform: Determine the first partial result corresponding to the basic time interval starting from the start time. The first partial result is obtained by processing the business data set within the basic time interval as a whole. Multiple different quantity values ​​are determined, each quantity value indicating a corresponding number of unit times following the base time interval; For each of the stated quantity values, the following steps are performed in parallel: obtaining the set of business data within a unit time period corresponding to the stated quantity value; performing the overall processing on the set of business data to obtain the second partial result corresponding to the stated quantity within a unit time period; reusing the first partial result; and performing the overall processing on the second partial result incrementally to obtain the global result at the end time corresponding to the stated quantity value.

Images