Method, device and equipment for determining cumulative independent access volume based on flow-batch coordination

Abstract

The embodiments of the present specification disclose a method for determining the cumulative independent access volume based on stream-batch coordination, the method comprising: obtaining a user access stream through a stream data source; extracting user access data from the user access stream at a preset time interval to obtain a batch data source; creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated; creating a stream task corresponding to the current time period, executing the stream task to perform re-deduplication according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, and obtaining the cumulative independent access volume in the current time period.

Description

Technical Field

[0001] The present specification relates to the field of computer technology, and in particular to a method, device and apparatus for determining cumulative independent access volume based on flow-batch coordination. Background Art

[0002] In many cases, it is necessary to count the cumulative independent visits in order to better understand the practical effect through the cumulative independent visits. For example, a website needs to count the increase in new users, and an activity needs to count the participation of new users.

[0003] In the stream computing scenario, the cumulative independent visits are determined through the stream task. The stream task processes the access data of the current time period in real time to determine the cumulative independent visits of the current time period. However, the cumulative independent visits obtained by relying solely on the stream task may be inaccurate and difficult to provide users with reliable reference data.

[0004] Based on this, an accurate way to determine the cumulative number of unique visits is now needed. Summary of the invention

[0005] One or more embodiments of this specification provide a method, device and apparatus for determining cumulative independent access volume based on flow-batch coordination, which are used to solve the following technical problems:

[0006] In the stream computing scenario, the cumulative independent access volume obtained only by stream tasks may be inaccurate and difficult to provide users with reliable reference data.

[0007] One or more embodiments of this specification adopt the following technical solutions:

[0008] One or more embodiments of this specification provide a method for determining cumulative independent access volume based on stream-batch coordination, the method comprising:

[0009] Obtain user access flow through the flow data source;

[0010] Extracting user access data from the user access stream at preset time intervals to obtain a batch data source;

[0011] Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated;

[0012] A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

[0013] One or more embodiments of the present specification further provide a device for determining cumulative independent access volume based on flow-batch coordination, the device comprising:

[0014] An acquisition unit acquires user access flow through a flow data source;

[0015] An extraction unit, extracting user access data from the user access stream at a preset time interval to obtain a batch data source;

[0016] A dimension table determination unit, which creates a batch task according to the batch data source, executes the batch task to perform deduplication, and obtains a historical access dimension table that is at least partially deduplicated;

[0017] The accumulation unit creates a flow task corresponding to the current time period, executes the flow task to perform deduplication according to the current historical access dimension table and the flow data corresponding to the current time period in the user access flow, and obtains the accumulated independent access volume in the current time period.

[0018] One or more embodiments of the present specification further provide a device for determining cumulative independent access volume based on flow-batch coordination, including:

[0019] at least one processor; and,

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0022] Obtain user access flow through the flow data source;

[0023] Extracting user access data from the user access stream at preset time intervals to obtain a batch data source;

[0024] Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated;

[0025] A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

[0026] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0027] Obtain user access flow through the flow data source;

[0028] Extracting user access data from the user access stream at preset time intervals to obtain a batch data source;

[0029] Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated;

[0030] A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

[0031] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0032] 1. The embodiments of this specification obtain user access flows through a flow data source. The user access flows here are full data accumulated in real time. When the subsequent accumulated independent access volume is determined, the result can be more accurate and reliable based on the full data.

[0033] 2. The embodiments of this specification extract user access data from the user access flow at preset time intervals to obtain a batch data source, wherein the batch data source is the historical access data before the current time period. Dividing the data according to preset time intervals can better deduplicate the historical access data.

[0034] 3. The embodiment of this specification deduplicates batch data sources through batch tasks to obtain a deduplicated historical access dimension table. During this period, the historical access data is deduplicated in segments through batch tasks, which can minimize the data in the historical access dimension table, so that the cumulative independent access volume can be better determined later.

[0035] 4. The embodiment of this specification executes the flow task of the current time period to deduplicate based on the current historical access dimension table and the flow data corresponding to the current time period in the user access flow, so as to obtain the cumulative independent access volume in the current time period. This process is a collaborative processing of the flow task and the batch task, and the full access data composed of the deduplicated historical access data and the access data of the current time period is calculated to obtain a more accurate cumulative independent access volume in the current time period. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0037] Figure 1 A flowchart of a method for determining cumulative independent access volume based on flow-batch coordination provided in one or more embodiments of this specification;

[0038] Figure 2 A schematic diagram of the structure of a stream-batch integrated computing system framework provided for one or more embodiments of this specification;

[0039] Figure 3 A schematic diagram of the structure of a device for determining the cumulative independent access volume based on flow-batch coordination provided in one or more embodiments of this specification;

[0040] Figure 4 A schematic diagram of the structure of a device for determining the cumulative independent access volume based on stream-batch coordination provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0041] The embodiments of this specification provide a method, device and equipment for determining the cumulative independent access volume based on flow-batch coordination.

[0042] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0043] Cumulative unique visits refer to the number of new unique visits in a specific time period, which is obtained after deduplication. When deduplication is performed on cumulative unique visits, it is not only for the access data in a specific time period, but also for the historical access data before the specific time period.

[0044] The following example illustrates the calculation rules for cumulative independent visits:

[0045] For example, when counting the cumulative visits to website A, the same user who visits website A multiple times during the period can only be counted as once. If the specific time period is from 0:00 on September 1, 2021 to 12:00 on September 1, 2021, the cumulative independent visits are the new independent visits from 0:00 on September 1, 2021 to 12:00 on September 1, 2021. If user a has visited website A before 0:00 on September 1, 2021, and user a visits website A again at 10:00 on September 1, 2021 and 11:00 on September 1, 2021, since user a has visited website A before 0:00 on September 1, 2021, even if user a visits website A again at 10:00 on September 1, 2021 and 11:00 on September 1, 2021, user a cannot be counted in the cumulative independent visits.

[0046] Similarly, the cumulative visits to website A from 0:00 on September 1, 2021 to 12:00 on September 1, 2021 are counted. If user b has not visited website A before 0:00 on September 1, 2021, and user b visits website A at 8:00 on September 1, 2021 and 10:00 on September 1, 2021, user b can be counted in the cumulative independent visits, and the visit time when recorded in the cumulative independent visits can be recorded as 8:00 on September 1, 2021.

[0047] In the scenario of cumulative independent visits, it is necessary to calculate the cumulative independent visits for a specific time period based on the full amount of data after deduplication. In the scenario of stream computing, independent visits (not cumulative independent visits here) will be determined through stream computing, because when the stream computing engine starts calculating, there is no historical access data, resulting in the fact that some newly added data is not completely deduplicated, and it is impossible to provide users with accurate cumulative independent visits. Only the independent visits that are not deduplicated with historical access data can be calculated. Among them, stream computing is real-time processing of data streams.

[0048] The following example illustrates how to determine the number of independent visits using flow calculation:

[0049] 1) To calculate the number of unique visits from midnight to the current time in real time, you can use the following SQL command:

[0050] select DateUtil(event_time,'yyyyMMdd'),count(distinct user)

[0051] from page_visit

[0052] group by DateUtil(event_time,'yyyyMMdd');

[0053] 'yyyyMMdd' is the date, accurate to the day, and distinct user is a unique user. For example, 'yyyyMMdd' is September 1, 2021. The above SQL instruction means selecting unique users with a time of September 1, 2021 from the current user access stream and grouping the users. It can be seen that the SQL instruction can calculate the number of unique visits on that day, but due to the defects of stream calculation, historical access data is not considered, and the cumulative number of unique visits on that day cannot be obtained.

[0054] 2) Calculate the current cumulative visits per hour in real time. 0 o'clock...12 o'clock...24 o'clock, which can be completed through the following SQL command:

[0055] select DateUtil(event_time,'yyyyMMddHH'),count(distinct user)

[0056] from page_visit

[0057] group by DateUtil(event_time,'yyyyMMddHH');

[0058] 'yyyyMMdd' is the date, accurate to the hour. The rest is the same as the previous SQL command. It is impossible to get the cumulative number of unique visits per hour.

[0059] In view of the above-mentioned defects, the technical solution provided in this specification is described in detail below in conjunction with the accompanying drawings.

[0060] Figure 1 A flowchart of a method for determining cumulative independent visits based on stream-batch coordination is provided for one or more embodiments of this specification. The process can be executed by a stream-batch integrated computing system. Certain input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0061] S102, obtaining user access flow through the flow data source.

[0062] The stream data source can be the full amount of data accumulated in real time, that is, each time a user accesses, it is accumulated to the stream data source. The user access stream can include at least independent statistical dimensions and access time. Creating a user access stream can be completed through the following SQL instructions:

[0063] create table page_visit(

[0064] user STRING'user id',

[0065] event_time BIGINT'visit timestamp'

[0066] dt STRING 'day partition' );

[0068] In the above SQL instructions, the independent statistical dimension is the user ID, and page_visit is the user visit flow. When creating page_visit, it includes user id (user ID), visit timestamp (visit time) and day partition (time format separated by day).

[0069] S104, extracting user access data from the user access flow at a preset time interval to obtain a batch data source.

[0070] In the embodiment of the present specification, in order to ensure that the cumulative independent access volume is accurate, the batch data source needs to be extracted from the first user access data so that the cumulative independent access volume is obtained based on the full amount of data.

[0071] The preset time interval can be determined according to the actual situation to ensure that users within the preset time interval can extract reasonably. If the daily visits are lower than the preset value, the preset time interval can be set to 3 days. If the daily visits are higher than the preset value, the preset time interval can be set to 12 hours. If the daily visits are the preset value, the preset time interval can be set to 1 day.

[0072] S106, creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated.

[0073] The batch data source is the user access data of each preset time interval. It is necessary to deduplicate the user access data of each preset time interval to obtain the deduplicated historical access dimension table. The historical access dimension table is deduplicated at each preset time interval, but the user access data between the preset time intervals is not deduplicated. This operation does not affect the subsequent determination of the cumulative access volume, which will be explained in detail later.

[0074] At this time, the user access data between the preset time intervals can also be deduplicated. It only needs to compare the user access data between the preset time intervals and filter out duplicate user access data to complete the user access data between the preset time intervals.

[0075] Furthermore, creating a batch task based on a batch data source can be performed based on the current time period. The preset time intervals do not belong to the current time period. The user access data for each preset time interval is the historical access data before the current time period. The access data corresponding to the current time period is the time period for determining the cumulative independent access volume. The following is an example:

[0076] For example, at 12:00 on September 1, 2021, to calculate the cumulative visits from 0:00 on September 1, 2021 to 12:00 on September 1, 2021, all user access data before 0:00 on September 1, 2021 is needed, and all user access data before 0:00 on September 1, 2021 is stored in a batch data source, which is the user access data of each preset time interval, and the user access data of each preset time interval needs to be deduplicated separately.

[0077] It should be noted that different batch tasks are created according to user access data at each time interval preset in the batch data source, and the batch task processes the batch data source through batch calculation.

[0078] Furthermore, before executing to obtain a historical access dimension table that is at least partially deduplicated, the following steps need to be performed: obtain the independent statistical dimension corresponding to the cumulative independent access volume to be determined, and create a historical access dimension table that contains only the independent statistical dimension. The historical access dimension table is updated by executing batch tasks, and the multiple attribute dimensions corresponding to the user access flow include the independent statistical dimension.

[0079] The historical access dimension table has only independent statistical dimensions, which can save memory space and successfully deduplicate the user access data at each preset time interval. The historical access dimension table can be created using the following SQL command:

[0080] create table dim_history_visit(

[0081] user STRING'user id' );

[0083] In the above SQL instructions, the independent statistical dimension is the user ID.

[0084] When S106 is executed, batch tasks can be created according to the user access data corresponding to each preset time interval in the batch data source and the corresponding third deduplication SQL instruction, and each batch task can be executed to perform deduplication, so as to obtain a historical access dimension table with deduplication for each time interval.

[0085] Deduplication of user access data at each preset time interval can be accomplished by the following SQL instruction (third deduplication SQL instruction):

[0086] insert into dim_history_visit

[0087] select distinct user from page_visit where dt='${bizDate}'

[0088] At this time, the independent statistical dimension is the user dimension, dim_history_visit is the historical visit dimension table, and bizDate is the preset time interval. The meaning of this SQL instruction is to insert the historical visit dimension table and deduplicate the same users from the preset time intervals so that the access record of the same user is only retained once in each preset time interval.

[0089] S108, creating a stream task corresponding to the current time period, executing the stream task to perform deduplication according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, and obtaining the cumulative independent access volume in the current time period.

[0090] When executing to create a stream task corresponding to the current time period, the inquirer who receives the accumulated independent visits sends a first deduplication SQL instruction that is unrelated to the stream batch; the second deduplication SQL instruction is obtained by automatically rewriting the first deduplication SQL instruction; and according to the second deduplication SQL instruction, a stream task corresponding to the current time period is created.

[0091] The first deduplication SQL instruction is input by the user to calculate the cumulative independent visits in the current time period. The user may not know the second deduplication SQL instruction, and can also be rewritten into the second deduplication SQL instruction. The first deduplication SQL instruction can provide corresponding parameters for rewriting into the second deduplication SQL instruction. Of course, the user may also know the second deduplication SQL instruction, and the user directly inputs the first deduplication SQL instruction, which can save the user's time.

[0092] The first deduplication SQL instruction can be completed by the following SQL instruction:

[0093] select

[0094] deduplicate('max 1',user,event_time)

[0095] from

[0096] page_visit

[0097] group by

[0098] user

[0099] Among them, 'max 1', user, event_time in the above SQL instruction is used to limit the maximum value of the access time selected in the second deduplication SQL instruction, and page_visit is used to limit the user access flow in the current time period in the second deduplication SQL instruction.

[0100] It should be noted that by automatically rewriting the first deduplication SQL instruction to obtain the second deduplication SQL instruction, the third deduplication SQL instruction executed in S106 can also be obtained. At this time, the first deduplication SQL instruction can be rewritten into the second deduplication SQL instruction and the third deduplication SQL instruction to provide corresponding parameters.

[0101] In addition, the third deduplication SQL instruction can also be obtained by automatically rewriting the first deduplication SQL instruction when creating a batch task. Specifically, first obtain the preset time interval corresponding to the current time period as the latest batch task scheduling time; then automatically rewrite the first deduplication SQL instruction according to the latest batch task scheduling time to obtain the third deduplication SQL instruction.

[0102] When S108 is executed, the flow data of the current time period is connected with the current historical access dimension table; and in the process of connection, the flow data of the current time period and the current historical access dimension table are dimensionally aligned according to the independent statistical dimension to obtain the connection result; the user records that exist in the flow data of the current time period but do not exist in the current historical access dimension table are filtered out from the connection result; and the execution of the flow task is completed by deduplicating the filtered results.

[0103] When the execution step deduplicates the filtered results and completes the execution of the flow task, the user access records contained in the filtered results can be grouped according to independent statistical dimensions to obtain one or more access record groups; the user access records in each access record group are individually numbered to obtain the record row number of each user access record; the same record row number is used to filter in each access record group, so that each access record group after filtering retains only one user access record; and the execution of the flow task is completed based on the retained user access records.

[0104] To calculate the cumulative number of unique visits in the current time period, you can use the following SQL instructions:

[0105] select

[0106] user,event_time

[0107] from (

[0109] select

[0110] *

[0111] Row_Number()over(partiton by user order by event_timedesc)as rn

[0112] from(--Dimension table association, filtering historical cold data

[0113] select*from page_visit p left join dim_history_visit h

[0114] on p.user=h.user where h.user is null )

[0116] )t where t.rn=1

[0117] Among them, the independent statistical dimension is the user dimension, Row_Number()over(partition by user orderby event_time) is used to process the access data in the current time period, which is used to indicate grouping according to the user dimension and sorting by access time within the group, desc indicates sorting the access time in descending order, that is, sorting from large to small, as rn indicates recording the sorted result as rn, select*from page_visit p left join dim_history_visit honp.user=h.user where h.user is null is used to associate the user access data of the current time period with the dimension table and filter the historical access data, page_visit p indicates recording the access data of the current time period as p, dim_history_visit h indicates recording the historical access dimension table as h, left join indicates left joining the access data of the current time period with the historical access dimension table, on p.user=h.user means that the user dimension in the access data of the current time period is aligned with the user dimension in the historical access dimension table first. select* means that based on the user dimension, the user access records that do not exist in the historical access dimension table are selected from the access data of the current time period. t means that the access stream data of the current time period filtered according to the historical access data is recorded as t. where t.rn=1 means that the access stream data of the current time period is filtered again. The access data of the same user dimension can be grouped and the corresponding row numbers can be generated through Row_Number(). At this time, only the access data with row number 1 in each group needs to be selected for accumulation.

[0118] It should be noted that in S106, the user access data between the preset time intervals is not deduplicated, and it will not affect the accuracy of the calculated cumulative independent access volume. This is because in S108, it is only necessary to determine that the access record corresponding to the independent statistical dimension in the user access flow of the current time period does not appear in the historical access dimension table. If the independent statistical dimension is the user dimension, it is only necessary to determine that a certain user ID in the user access flow of the current time period has not appeared in the historical access dimension table, so that it can be determined that the user ID can be accumulated.

[0119] The above scheme is to calculate the cumulative visits for the current time period. The condition for generating the current time period may be that the user issues an instruction at any time. The embodiment of this specification may also calculate the cumulative visits for each time period. The following is an explanation of this situation:

[0120] According to the user access records from the user access flow in sequence, a flow task for each preset time interval is created. The flow task is created at this time to realize the calculation of the cumulative access volume for each preset time interval. Each time interval here can represent the current time period of the above step, which is different from the time interval of S104. Then, the corresponding user access flow is obtained through the flow task corresponding to each preset time interval, and the corresponding historical access dimension table that is at least partially deduplicated is obtained through the batch task corresponding to each preset time interval. Then, according to the user access flow and the historical access dimension table corresponding to each preset time interval, the flow task for each preset time interval is executed to perform deduplication, so as to obtain the cumulative independent access volume within each preset time interval, and the cumulative independent access volume for each preset time interval is stored in the cumulative independent access volume database.

[0121] Furthermore, the embodiments of this specification can be applied in user conversion promotion activities. By calculating the cumulative independent visits, the situation of new users can be well understood, the effect of promotion activities can be understood, and a reference can be provided for the direction of subsequent activities.

[0122] The following is a detailed description of the implementation promotion activities:

[0123] Determine a user conversion promotion activity starting from a first specified time, and receive a user conversion effect query instruction at a second specified time, wherein the second specified time is later than the first specified time, and the second specified time and the first specified time contain multiple preset time intervals; parse the user conversion effect query instruction to determine the time interval required for query, the time interval includes at least one preset time interval; call the cumulative independent visit volume database to obtain the cumulative independent visit volume corresponding to the time interval to determine the user conversion data.

[0124] Through the above operations, the cumulative independent visits in any interval between the first specified time and the second specified time can be queried to determine the user conversion data. The user conversion data can also indirectly reflect the effect of the promotion activities and provide a reference for whether to change subsequent promotion activities.

[0125] The embodiments of this specification implement collaboration between streams and batches based on the historical access dimension table, that is, batch tasks can be used to generate data on user dimensions that have been accessed historically.

[0126] The embodiments of this specification utilize the stream-batch integrated computing system framework and can automatically disassemble the historical access dimension table to generate batch tasks, thereby filtering the historical access data and ensuring that the cumulative independent access volume can be calculated in real time, thus overcoming the problem that the stream task engine cannot store historical access data. The stream-batch integrated computing system framework is described in detail below:

[0127] Figure 2 A structural diagram of a stream-batch integrated computing system framework provided for one or more embodiments of the present specification, in which a stream data source generates a stream task, and a batch data source generates a batch task. When stream-and-batch-independent SQL instructions are generated, they can be automatically rewritten into stream tasks and batch tasks. After the stream task is executed, the cumulative independent access volume can be calculated and stored uniformly.

[0128] Figure 3 A schematic diagram of the structure of a device for determining cumulative independent access volume based on stream-batch coordination provided for one or more embodiments of this specification, the device includes: an acquisition unit 302, an extraction unit 304, a dimension table determination unit 306 and an accumulation unit 308.

[0129] The acquisition unit 302 acquires the user access flow through the flow data source;

[0130] The extraction unit 304 extracts user access data from the user access stream at a preset time interval to obtain a batch data source;

[0131] The dimension table determining unit 306 creates a batch task according to the batch data source, executes the batch task to perform deduplication, and obtains a historical access dimension table that is at least partially deduplicated;

[0132] The accumulation unit 308 creates a flow task corresponding to the current time period, executes the flow task to perform deduplication according to the current historical access dimension table and the flow data corresponding to the current time period in the user access flow, and obtains the accumulated independent access volume in the current time period.

[0133] Furthermore, before the dimension table determining unit 306 executes to obtain at least a partially deduplicated historical access dimension table, the apparatus further includes:

[0134] The dimension acquisition unit 310 acquires the independent statistical dimension corresponding to the cumulative independent visits to be determined;

[0135] The dimension table creation unit 312 creates a historical access dimension table including only independent statistical dimensions. The historical access dimension table is updated by executing batch tasks. Multiple attribute dimensions corresponding to user access flows include independent statistical dimensions.

[0136] Further, when the accumulation unit 308 creates a flow task corresponding to the current time period, it is specifically used to:

[0137] The inquirer receiving the accumulated independent access volume sends the first deduplication SQL instruction that is irrelevant to the flow and batch;

[0138] By automatically rewriting the first deduplication SQL instruction, a second deduplication SQL instruction is obtained;

[0139] According to the second deduplication SQL instruction, a stream task corresponding to the current time period is created.

[0140] Further, when the dimension table determining unit 306 creates a batch task according to the batch data source, executes the batch task to perform deduplication, and obtains at least a partially deduplicated historical access dimension table, it is specifically used to:

[0141] Create batch tasks respectively according to the user access data corresponding to each preset time interval in the batch data source and the corresponding third deduplication SQL instruction;

[0142] Execute each batch of tasks to deduplicate and obtain a historical access dimension table with deduplication separately performed for each time interval.

[0143] Furthermore, when the dimension table determination unit 306 executes the batch creation task, it is specifically used to:

[0144] Get the preset time interval corresponding to the current time period as the latest batch task scheduling time;

[0145] According to the latest batch task scheduling time, the first deduplication SQL instruction is automatically rewritten to obtain a third deduplication SQL instruction.

[0146] Further, when the accumulation unit 308 executes the stream task to perform re-deduplication according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, it is specifically used to:

[0147] Connect the stream data of the current time period with the current historical access dimension table;

[0148] During the connection process, the flow data of the current time period is aligned with the current historical access dimension table according to the independent statistical dimension to obtain the connection result;

[0149] In the connection results, filter out user records that exist in the stream data of the current time period but do not exist in the current historical access dimension table;

[0150] The execution of the flow task is completed by removing duplicates from the filtered results.

[0151] Further, when the accumulating unit 308 performs deduplication on the screening results and completes the execution of the convection task, it is specifically used to:

[0152] According to the independent statistical dimension, each user access record contained in the filtered result is grouped to obtain one or more access record groups;

[0153] The user access records in each access record group are numbered separately to obtain the record row number of each user access record;

[0154] Filter each access record group with the same record row number, so that each access record group after filtering retains only one user access record;

[0155] Complete the execution of the flow task based on the retained user access records.

[0156] Furthermore, the device also includes:

[0157] The flow task creation unit 314 creates flow tasks for each preset time interval according to each user access record from the user access flow in sequence;

[0158] The access flow acquisition unit 316 acquires the corresponding user access flow through the preset flow tasks corresponding to each time interval, and acquires the corresponding at least partially deduplicated historical access dimension table through the preset batch tasks corresponding to each time interval;

[0159] The cumulative storage unit 318 executes the preset flow tasks for each time interval to deduplicate according to the user access flow and historical access dimension table corresponding to each preset time interval, obtains the cumulative independent access volume within each preset time interval, and stores the cumulative independent access volume for each preset time interval in the cumulative independent access volume database.

[0160] Furthermore, the device also includes:

[0161] The activity determination unit 320 determines a user conversion promotion activity starting from the first specified time;

[0162] The instruction receiving unit 322 receives a user conversion effect query instruction at a second specified time, wherein the second specified time is later than the first specified time, and there are multiple preset time intervals between the second specified time and the first specified time;

[0163] The query time determination unit 324 analyzes the user conversion effect query instruction and determines the time interval required for the query, where the time interval includes at least one preset time interval;

[0164] The data determination unit 326 calls the cumulative independent visit volume database to obtain the cumulative independent visit volume corresponding to the time interval to determine the user conversion data.

[0165] Figure 4 A schematic diagram of a structure of a device for determining cumulative independent access volume based on flow-batch coordination provided for one or more embodiments of this specification includes:

[0166] at least one processor; and,

[0167] a memory communicatively connected to at least one processor; wherein,

[0168] The memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable the at least one processor to:

[0169] Obtain user access flow through the flow data source;

[0170] Extracting user access data from the user access stream at a preset time interval to obtain a batch data source;

[0171] Creating a batch task according to a batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated;

[0172] Create a stream task corresponding to the current time period, execute the stream task to deduplicate based on the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, and obtain the cumulative independent access volume in the current time period.

[0173] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0174] Obtain user access flow through the flow data source;

[0175] Extracting user access data from the user access stream at preset time intervals to obtain a batch data source;

[0176] Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated;

[0177] A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

[0178] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0179] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0180] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0181] For the convenience of description, the above device is described by dividing it into various units according to its functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0182] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0183] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0187] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0188] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0189] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0190] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0191] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0192] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0193] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for determining cumulative independent access volume based on flow-batch coordination, the method comprising: Obtain user access streams through stream data sources. The stream data source can be the full amount of data accumulated in real time, that is, each user access is accumulated to the stream data source; Extracting user access data from the user access stream at preset time intervals to obtain a batch data source; Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated; A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

2. According to the method of claim 1, before obtaining the at least partially deduplicated historical access dimension table, the method further comprises: Obtaining an independent statistical dimension corresponding to the cumulative independent visits to be determined; A historical access dimension table is created, the dimension of which is only the independent statistical dimension. The historical access dimension table is updated by executing the batch task. The multiple attribute dimensions corresponding to the user access flow include the independent statistical dimension.

3. According to the method of claim 1, the creating a stream task corresponding to the current time period specifically comprises: The inquirer receiving the accumulated independent access volume sends a first deduplication SQL instruction irrelevant to the flow and batch; By automatically rewriting the first deduplication SQL instruction, a second deduplication SQL instruction is obtained; According to the second deduplication SQL instruction, a stream task corresponding to the current time period is created.

4. The method according to claim 3, wherein the step of creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated, specifically comprises: Create batch tasks respectively according to the user access data corresponding to each preset time interval in the batch data source and the corresponding third deduplication SQL instruction; Execute each of the batch tasks to perform deduplication, and obtain a historical access dimension table with deduplication separately performed for each time interval.

5. The method according to claim 4, wherein the creating a batch task specifically comprises: Obtain the preset time interval corresponding to the current time period as the latest batch task scheduling time; The first deduplication SQL instruction is automatically rewritten according to the latest batch task scheduling time to obtain the third deduplication SQL instruction.

6. The method according to claim 2, wherein the step of executing the stream task to perform re-deduplicate based on the current historical access dimension table and the stream data corresponding to the current time period in the user access stream comprises: Connecting the stream data of the current time period with the current historical access dimension table; During the connection process, dimension alignment is performed on the stream data of the current time period and the current historical access dimension table according to the independent statistical dimension to obtain the connection result; Filtering out user records that exist in the stream data of the current time period but do not exist in the current historical access dimension table from the connection result; The execution of the stream task is completed by removing duplicates from the screening results.

7. The method according to claim 6, wherein the performing of deduplication on the screening results to complete the execution of the stream task specifically comprises: Grouping the user access records contained in the screening results according to the independent statistical dimension to obtain one or more access record groups; The user access records in each access record group are individually numbered to obtain the record row number of each user access record; Filtering each of the access record groups using the same record row number so that each of the access record groups after filtering retains only one user access record; The execution of the stream task is completed according to the retained user access record.

8. The method according to claim 1, further comprising: Creating flow tasks for each preset time interval according to each user access record from the user access flow in sequence; Obtain corresponding user access flows through preset stream tasks corresponding to each time interval, and obtain corresponding historical access dimension tables that are at least partially deduplicated through preset batch tasks corresponding to each time interval; According to the user access flow and historical access dimension table corresponding to each preset time interval, the preset flow task of each time interval is executed to deduplicate, the cumulative independent access volume within each preset time interval is obtained, and the cumulative independent access volume of each preset time interval is stored in the cumulative independent access volume database.

9. The method according to claim 8, further comprising: Determine a user conversion promotion activity starting from a first specified moment; receiving a user conversion effect query instruction at a second designated time, wherein the second designated time is later than the first designated time, and a plurality of the preset time intervals are included between the second designated time and the first designated time; Parsing the user conversion effect query instruction to determine a time interval required for querying, wherein the time interval includes at least one of the preset time intervals; The cumulative independent visit volume database is called to obtain the cumulative independent visit volume corresponding to the time interval to determine the user conversion data.

10. A device for determining cumulative independent access volume based on flow-batch coordination, the device comprising: The acquisition unit acquires the user access stream through the stream data source. The stream data source can be the full amount of data accumulated in real time, that is, each user access is accumulated to the stream data source; An extraction unit, extracting user access data from the user access stream at a preset time interval to obtain a batch data source; A dimension table determination unit, which creates a batch task according to the batch data source, executes the batch task to perform deduplication, and obtains a historical access dimension table that is at least partially deduplicated; The accumulation unit creates a flow task corresponding to the current time period, executes the flow task to perform deduplication according to the current historical access dimension table and the flow data corresponding to the current time period in the user access flow, and obtains the accumulated independent access volume in the current time period.

11. The device according to claim 10, before the dimension table determination unit executes obtaining the at least partially deduplicated historical access dimension table, the device further comprises: A dimension acquisition unit, which acquires an independent statistical dimension corresponding to the cumulative independent visits to be determined; A dimension table creation unit creates a historical access dimension table including only the independent statistical dimension, wherein the historical access dimension table is updated by executing the batch task, and the multiple attribute dimensions corresponding to the user access flow include the independent statistical dimension.

12. The device according to claim 10, when the accumulation unit executes the creation of the flow task corresponding to the current time period, is specifically used to: The inquirer receiving the accumulated independent access volume sends a first deduplication SQL instruction irrelevant to the flow and batch; By automatically rewriting the first deduplication SQL instruction, a second deduplication SQL instruction is obtained; According to the second deduplication SQL instruction, a stream task corresponding to the current time period is created.

13. The apparatus according to claim 12, wherein when the dimension table determination unit creates a batch task according to the batch data source, executes the batch task to perform deduplication, and obtains a historical access dimension table that is at least partially deduplicated, the dimension table is specifically configured to: Create batch tasks respectively according to the user access data corresponding to each preset time interval in the batch data source and the corresponding third deduplication SQL instruction; Execute each of the batch tasks to perform deduplication, and obtain a historical access dimension table with deduplication separately performed for each time interval.

14. The device according to claim 13, when the dimension table determination unit executes the batch creation task, is specifically used to: Obtain the preset time interval corresponding to the current time period as the latest batch task scheduling time; The first deduplication SQL instruction is automatically rewritten according to the latest batch task scheduling time to obtain the third deduplication SQL instruction.

15. The device according to claim 11, when the accumulating unit executes the stream task to perform re-deduplicate based on the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, it is specifically used to: Connecting the stream data of the current time period with the current historical access dimension table; During the connection process, dimension alignment is performed on the stream data of the current time period and the current historical access dimension table according to the independent statistical dimension to obtain the connection result; Filtering out user records that exist in the stream data of the current time period but do not exist in the current historical access dimension table from the connection result; The execution of the stream task is completed by removing duplicates from the screening results.

16. The device according to claim 15, when the accumulating unit performs deduplication on the screening result to complete the execution of the stream task, is specifically used to: Grouping the user access records contained in the screening results according to the independent statistical dimension to obtain one or more access record groups; The user access records in each access record group are individually numbered to obtain the record row number of each user access record; Filtering each of the access record groups using the same record row number so that each of the access record groups after filtering retains only one user access record; The execution of the stream task is completed according to the retained user access record.

17. The apparatus according to claim 10, further comprising: A flow task creation unit, which creates flow tasks for each preset time interval according to each user access record from the user access flow in sequence; An access flow acquisition unit, which acquires corresponding user access flows through flow tasks corresponding to preset time intervals, and acquires corresponding at least partially deduplicated historical access dimension tables through batch tasks corresponding to preset time intervals; The cumulative storage unit executes the preset flow tasks of each time interval to deduplicate according to the user access flow and historical access dimension table corresponding to each preset time interval, obtains the cumulative independent access volume within each preset time interval, and stores the cumulative independent access volume of each preset time interval in the cumulative independent access volume database.

18. The apparatus according to claim 17, further comprising: an activity determination unit, determining a user conversion promotion activity starting from a first specified time; An instruction receiving unit receives a user conversion effect query instruction at a second specified time, wherein the second specified time is later than the first specified time, and the second specified time and the first specified time contain a plurality of the preset time intervals; A query time determination unit, which parses the user conversion effect query instruction and determines a time interval required for querying, wherein the time interval includes at least one of the preset time intervals; The data determination unit calls the cumulative independent visit volume database to obtain the cumulative independent visit volume corresponding to the time interval to determine the user conversion data.

19. A device for determining cumulative independent access volume based on flow-batch coordination, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Obtain user access streams through stream data sources. The stream data source can be the full amount of data accumulated in real time, that is, each user access is accumulated to the stream data source; Extracting user access data from the user access stream at preset time intervals to obtain a batch data source; Creating a batch task according to the batch data source, executing the batch task to perform deduplication, and obtaining a historical access dimension table that is at least partially deduplicated; A stream task corresponding to the current time period is created, and according to the current historical access dimension table and the stream data corresponding to the current time period in the user access stream, the stream task is executed to perform deduplication again to obtain the cumulative independent access volume in the current time period.

Images