Scalable window aggregator

Abstract

A method executed by a computer is disclosed. The method includes receiving a data set including one or more data fields. The computer can then receive instructions for calculating aggregated features of the data set. The computer can then determine operators and windows based on the aggregated features to be calculated. The computer can then include the computer processing the data set. The computer can then apply the operator with the window to a processed data set to calculate the aggregated feature of the data set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a PCT application claiming priority to U.S. Provisional Application No. 63 / 355,938, filed on June 27, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Aggregate features of data sets are used for training and real-time use of many machine learning models. Window aggregate functions can be used to calculate aggregate features, such as the sum, average, minimum, or maximum of a large data set within a window. Aggregate features calculated by windowed aggregate functions can be used for training and real-time use of artificial intelligence models. However, many current implementations of window aggregate functions are not efficient when applied to large data sets. The runtime of calculating aggregate features of large data sets can be improved. Further, skew in the data set or missing values ​​in the data set adversely affect the runtime of aggregate feature calculation.

[0004] Embodiments of the present disclosure address this and other problems, individually and collectively. Summary of the invention

[0005] One embodiment of the present disclosure includes a method. The method includes: storing a data set including N data measurement vectors, each data measurement vector including a time field and a data field, wherein N is at least 100,000, wherein the data field of each data measurement vector includes a measurement value of an attribute of a network system corresponding to a time value; receiving a command for calculating an aggregate feature vector of an aggregate feature of the data set, wherein the aggregate feature vector includes a corresponding aggregate value of the data field of each sliding time window across a plurality of sliding time windows of the data set; determining an operator for determining the corresponding aggregate value and a window size for the aggregate feature to be calculated; for a first time window, using the measurement value of a first data measurement vector having a time field within the first time window, determining the first aggregate value using an operator ; advancing the window pointer to define a second time window, wherein the second time window overlaps the first time window, and wherein the second time window includes a second data measurement vector; identifying a first portion of the first measurement vector that is not within the second time window; identifying a second portion of the second measurement vector that is not within the first time window; determining a second aggregate value using an operator by: removing the measurement value of the first portion of the first measurement vector from the first aggregate value; and including the measurement value of the second portion of the second measurement vector in the first aggregate value; and repeating the advancement of the window pointer for consecutive time windows, and repeating the identification of data measurement vectors for removal and inclusion, so that the operator determines consecutive aggregate values ​​for consecutive time windows in multiple sliding time windows.

[0006] Another embodiment of the present invention includes a computer. The computer includes: a processor; and a non-transitory computer-readable medium, the non-transitory computer-readable medium containing instructions that can be executed by the processor to perform operations including the following: storing a data set including N data measurement vectors, each data measurement vector including a time field and a data field, where N is at least 100,000, wherein the data field of each data measurement vector includes a measurement value of an attribute of a network system corresponding to a time value; receiving a command for calculating an aggregate feature vector of an aggregate feature of the data set, wherein the aggregate feature vector includes a corresponding aggregate value of the data field of each sliding time window across a plurality of sliding time windows of the data set; determining an operator for determining the corresponding aggregate value and a window size for the aggregate feature to be calculated; for a first time window, using a time field having a time field within the first time window; The invention relates to a method for determining a first aggregate value of a first data measurement vector of a field, wherein the first aggregate value is determined using an operator; advancing a window pointer to define a second time window, wherein the second time window overlaps with the first time window, and wherein the second time window includes a second data measurement vector; identifying a first portion of the first measurement vector that is not within the second time window; identifying a second portion of the second measurement vector that is not within the first time window; determining a second aggregate value using an operator by: removing the measurement value of the first portion of the first measurement vector from the first aggregate value; and including the measurement value of the second portion of the second measurement vector in the first aggregate value; and repeating the advancement of the window pointer for consecutive time windows, and repeating the identification of data measurement vectors for removal and inclusion, so that the operator determines consecutive aggregate values ​​for consecutive time windows in multiple sliding time windows.

[0007] A better understanding of the nature and advantages of embodiments of the present invention may be obtained by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A block diagram of a data set with overlapping windows is shown according to an embodiment.

[0009] Figure 2 A block diagram of an interaction dataset according to an embodiment is shown.

[0010] Figure 3 A block diagram of a window and an expanded linked list is shown according to an embodiment.

[0011] Figure 4A and Figure 4B A block diagram of aggregate feature computation according to an embodiment is shown.

[0012] Figure 5 A block diagram showing two and aggregate feature calculations performed using a 20 second time window and a 30 second time window, according to an embodiment.

[0013] Fig. 6A and Figure 6B A block diagram of minimum and maximum aggregate feature calculations according to an embodiment is shown.

[0014] Figure 7 A block diagram of a system architecture according to an embodiment is shown.

[0015] Figure 8 A flow chart of a method according to an embodiment is shown.

[0016] Fig. 9 A block diagram of an exemplary computer system is shown in accordance with an embodiment. DETAILED DESCRIPTION

[0017] Window aggregation is the computation of aggregate features of a data set. Aggregate features are used in many machine learning and artificial intelligence models that can capture both short-term and long-term behaviors of a data set. The implementation uses an existing distributed system framework to sort large data sets and applies a shared computing framework to perform aggregate feature computations. The computation of aggregate features operates on sliding windows of the data set, where the sliding windows overlap in time.

[0018] The embodiment shares computation when performing multiple operations to compute multiple aggregate features of a data set. In the process of computing multiple aggregate features of a data set, results from previous windows can be reused so as to use less memory and perform computations more efficiently. For example, while sliding a window across a data set, the aggregate value of a previous overlapping window can be used to determine the aggregate value of the next window by: (1) adding the value of new data in the data set that appears in the next window but is not in the previous window, and (2) subtracting the value of data that is in the previous window but not in the next window. In this way, the embodiment provides an efficient method for performing computations using a sliding window. This computational method makes the computational complexity of most aggregate functions independent of the window size or the number of data elements in the window. In addition, the embodiment can share computational results to compute additional aggregate features of a data set. For example, if an aggregate feature of a data set including sums and counts has been computed, the average aggregate feature of the data set can be simply computed using sums and counts.

[0019] The embodiment shares data when performing multiple operations to calculate multiple aggregate features of a data set. When calculating multiple aggregate features of a data set, multiple windows can be applied to the same data. For example, when the embodiment calculates multiple aggregate features of a data set, two windows can be applied to the same piece of data. Each of the two windows can include a window pointer, which can point to the specific piece of data to which the window is applied. By using window pointers, the same copy of the data set can be reused for two calculations, thereby providing a significant reduction in memory requirements for operations with larger window sizes.

[0020] I. Window Aggregate Functions

[0021] A window aggregate function is a function that can be used to compute aggregate features (e.g., sum, count, average, minimum, maximum, distinct count, etc.) of a data set of events (e.g., interactions) by applying a window (e.g., a time interval) to the data set. For example, for a time series of interactions that occurred within a 24-hour period, a sliding window that implements aggregate feature computation can be applied to the time series. The sliding window can be used to compute the total amount of interactions in each 10-minute window of the 24-hour period. Assuming that the time series holds data for every minute of the 24-hour period, the sliding window can compute a total of 1431 features (e.g., 1431 = total minutes - (window size + 1) = (24 hours * 60 minutes) - 10-minute window + 1). Many of these windows contain overlapping timestamps. For example, a first window may contain data from 0 to 10 minutes, and a second window may contain data from 1 to 11 minutes, meaning that both windows contain data from 1 to 10 minutes. Partly due to this overlap, the implementation of window aggregate functions is not efficient when applied to large data sets. The runtime and memory requirements of such computations prove to be cumbersome.

[0022] A. Exemplary Datasets

[0023] Figure 1 1 shows a block diagram of a data set 100 with overlapping windows according to an embodiment. A first window 101, a second window 102, and a third window 103 may be applied to the data set 100. Each of the three windows may be a sliding window and have a size of three, and as Figure 1As shown, the windows are overlapping. The first window 101 includes timestamps 1 to 3, the second window 102 includes timestamps 2 to 4, and the third window 103 includes timestamps 3 to 5. As described above, many implementations of window aggregation functions are not efficient. For example, to calculate the sum of the values ​​in the first window 101, the three values ​​associated with timestamps 1 to 3 are summed, to calculate the sum of the values ​​in the second window 102, the three values ​​associated with timestamps 2 to 4 are summed, and to calculate the sum of the values ​​in the third window 103, the three values ​​associated with timestamps 3 to 5 are summed.

[0024] Adjacent windows include two-thirds of the same timestamps (e.g., the first window 101 and the second window 102 include timestamps 2 and 3, and the second window 102 and the third window 103 include timestamps 2 and 4). Therefore, the calculation of the sum of the adjacent windows includes two-thirds of the same elements. As the number of windows increases, the amount of repeated calculations also increases. In addition, several implementations of window aggregation functions require unique data structures, which means that the data set 100 is copied a number of times equal to the number of windows. For very large data sets, as the window slides across the data set 100, the amount of repeated calculations and the number of copies of the data set 100 increase significantly.

[0025] The window aggregation function can calculate the desired aggregate feature by applying an operator with a window to the data set. The operator can implement the calculation of the aggregate feature (for example, a sum operator, a count operator, an average operator, a minimum operator, a maximum operator, a different count operator, etc.). The window can implement the time interval for which the aggregate feature is to be calculated (for example, 30 seconds, 1 minute, 10 minutes, 15 minutes, 30 minutes, 1 hour, 4 hours, etc.).

[0026] Operators with appropriate windows can then be applied to the data set to calculate the desired aggregate features of the data set. For example, data set 100 can be a data set of interactions that occurred during a week that can be used (although only 13 timestamps are shown). Aggregate features of the data set, such as the sum of interactions in the last hour, can be calculated. In order to calculate such aggregate features of data set 100, a summation operator with a window of size 1 hour (for example, assuming that the timestamps in data set 100 are minutes, the window will have a size of 60) can be applied to the interaction value data field of data set 100. The window can initially include the first hour block of data set 100, and calculate the sum of interactions in this first hour block. After the sum of the first hour has been calculated, the window can be slid to the next timestamp to calculate the sum of interactions in the second hour block. The process can be repeated until the sum of each hour block of the data set is calculated. The embodiment provides an efficient method for performing such calculations.

[0027] Scalable window aggregators can be used to compute aggregate features of a data set. Scalable window aggregators provide a method for computing aggregate features of a data set such that the computational complexity of most aggregate functions is independent of the window size or the number of data elements in the window.

[0028] B. Interaction Dataset

[0029] Figure 2 A block diagram of an interaction data set 200 according to an embodiment is shown. The interaction data set 200 may include events sorted by timestamps stored in a timestamp 210 data row. The interaction data set 200 may include data of events occurring during the timestamp. An example of an interaction data set 200 may be a transaction data set that records the value of a transaction occurring at a timestamp in the timestamp 210 data row, a count of approved interactions occurring at a timestamp in an approval count row 230, a count of rejected interactions occurring at a timestamp in a rejection count row 240, and an identifier (e.g., name, email address, phone number, etc.) of one or more interaction parties in an identifier row 250. Data sets similar to the interaction data set 200 may include a traffic data set that records the amount of cars passing through a toll booth, a network traffic data set that records the amount of network traffic requests to a website, and the like. The interaction data set 200 may have any length, such as 100, 1000, 10000, 20000, 50000, 100000, 500000, 1 million, 1 billion, or more.

[0030] Such a dataset may store data for millions of interactions that occur daily for multiple entities (such as merchants, merchant categories, zip codes, etc.). Interaction dataset 200 corresponds to one entity. For example, a payment processing network may process hundreds of millions of transactions daily and store more data for each of these transactions in a dataset similar to interaction dataset 200. When computing aggregate features for such a dataset, a window is slid across interaction dataset 200 to determine a first aggregate feature value. The window is slid across each subsequent timestamp of the data to compute subsequent aggregate feature values ​​for different window spans. However, as the window is slid across subsequent timestamps, the window spans overlap to a large extent.

[0031] The interaction data set 200 can be processed to improve the efficiency of the aggregate feature calculation. The distributed system can distribute the data set to various processing nodes so that the calculation can only parse the relevant data (i.e., the calculation of the average interaction value does not require knowing the identifier of the transaction party). An example of such a distributed system includes the SPARK framework. The SPARK framework can be run to distribute parts of the data set to different processing nodes. For example, the SPARK framework or some other distributed system can distribute the interaction data set 200 to different processing nodes based on timestamp values ​​so that the computing load is evenly distributed (e.g., by mapping each column with a timestamp modulated by the total number of processing nodes). By evenly distributing the calculations, no one processing node will have a longer runtime than any other node because they all process the same amount.

[0032] However, some distributed systems distribute all null values ​​in a data set to the same processing node, which causes this processing node to have an abnormally long runtime. For example, when a table is distributed to processing nodes, each time the Spark framework identifies a null value in the table, the Spark framework will send each null value to a single processing node. For large data sets that may contain a large number of null values, such as a transaction data set, this may cause this single processing node to have a very long runtime.

[0033] To address this, null values ​​in a data set can be first handled by replacing them with identifiable random values. In doing so, calculations are more evenly distributed, which results in a reduction in data skew caused by null values. For example, if aggregate features are calculated for a specific email (e.g., johndoe@email.com), some entries in the data set may have null values ​​in the email data field. Each null value can start with an identifiable random value. For example, if there are a total of 100 processing nodes, a random number between 1 and 100 can be appended to the beginning of the null value so that the null value is mapped to a random processing node in each of the total 100 processing nodes. Aggregate features can then be calculated, and operators on each of the processing nodes can be configured to identify null entries by identifiable random values, and ignore calculations for each null entry.

[0034] II. Scalable Window Aggregator

[0035] The scalable window aggregator provides a framework for computing aggregate features of a dataset. Examples of such datasets may include Figure 2 and Figure 3 The data set shown in . The data set shown previously may be in some other data structure.

[0036] A. Exemplary Data Structures

[0037] Figure 3 A block diagram of a window and expanded linked list 300 is shown according to an embodiment. Figure 3 An example data type that can be used by a scalable window aggregator to calculate aggregate features of a data set is shown. Other data types that can be used by a scalable window aggregator include linked lists, arrays, stacks, queues, or any other linear data structure. Different data types can provide different benefits. For example, the use of an expanded linked list provides a data type under which, in addition to low overhead for memory usage and memory allocation, it is simple to insert elements at the tail end and remove elements from the head end. The expanded linked list 300 can include a first node 301, a second node 302, and a third node 303 that can store data fields of an input data set. For example, the nodes of the expanded linked list 300 can store value data fields of an interactive data set 200. In some examples, the input data set can be processed by a distributed system to improve efficiency and reduce memory requirements for performing calculations, and the resulting processed input data set can be an expanded linked list 300.

[0038] Each node in the linked list 300 of expansion can have a tail pointer. The tail pointer of the node points to the subsequent node. For example, the tail pointer 312 of the first node 301 can point to the rightmost block away from the first node 301 and point to the second node 302, the tail pointer 314 of the second node 302 can point to the third node 303, and the tail pointer 316 of the third node 303 can point to a node that does not currently exist. When a new node is to be added to the linked list 300 of expansion (for example, more data of a data set is received), the tail pointer of the node (for example, the tail pointer 316 of the third node 303) can be retrieved, and the tail pointer can be set to point to the new node. In some implementations, the node of the linked list of expansion can also include a head pointer in addition to the tail pointer. However, if only the tail pointer is used, automatic memory release can be achieved by recollecting the node after reaching the tail pointer.

[0039] In contrast to some current implementations that store data within each window, a window can include a window pointer to a block of nodes. This reduces the memory requirements for performing computations, as when a new window is added, the data does not have to be copied to the new window, and instead the new window can simply be applied to the already existing expanded linked list. Figure 3 In the example of , the first window pointer 306 of the first window may point to the first block of the first node 301. The second window pointer 308 may correspond to the second time window. An example of the second time window is described as follows.

[0040] As a first example, a first window pointer 306 of a first window may slide over the data in the first node 301 until it reaches the tail pointer 312 of the first node 301. For example, the first node 301 may store data having a timestamp between 1 and 100, the second node 302 may store data having a timestamp between 101 and 200, and the third node 303 may store data having a timestamp between 201 and 300. The first window pointer 306 may correspond to a time window of 10 seconds and may initially point to the first block of the first node 301. After calculating appropriate aggregate features, the window pointer of the first window pointer 306 may be moved to point to the second block of the first node 301. Appropriate aggregate features may be calculated and then the first window pointer 306 may be moved to point to the third block of the first node 301, continuing until the tail pointer 312 of the first node 301 is reached. After reaching the tail pointer 312, the first window pointer 306 may continue to the first timestamp of the second node 302, as shown by the second window pointer 308. Such an example is represented by FIG. 4A to FIG. 4B and Figure 5 A to Figure 5 B shows.

[0041] As a second example, the second window pointer 308 may be a window pointer of a second window having a different size than the first window pointer 306. One such example may be that the first window pointer 306 has a 10 second time window for calculating the sum of the last 10 seconds, and the second window pointer 308 has a 10 minute time window for calculating the sum of the last 10 minutes. Figure 5 Such an example is shown.

[0042] Contrary to many current implementations of aggregate feature computation, different windows can be applied to the same expanded linked list 300, and the expanded linked list 300 does not need to be reproduced or copied in any way. In this way, different windows can share data to reduce the memory requirements for computing multiple aggregate features of a data set, because the data held by the expanded linked list 300 does not need to be reproduced in any way.

[0043] B. and aggregate feature calculation

[0044] Each aggregate feature calculation can be implemented by applying an operator with a window to a data field. The calculation slides the data through the data field, applying the operator at each step. When the window is slid through the data field, the aggregate value of the previous window can be used to determine the aggregate value of the next window by adding the value of the new data in the next window but not in the previous window. Additionally, the value of the data in the previous window but not in the next window will be subtracted. In this way, calculations for sliding windows can be performed efficiently. This incremental calculation method makes the computational complexity of most aggregate functions independent of the window size. Figure 4 shows an example of using sliding window calculations and features.

[0045] Figure 4A and Figure 4B A block diagram of the computation of aggregated features according to an embodiment is shown. Figure 4A In the example, an aggregate feature of the sum of the interaction values ​​in the last 20 seconds may be desired. A summation operator may be selected, and a 20-second time window 403 may be selected. Data set 400 may include at least (value, timestamp) combinations of [(2, 10), (4, 20), (3, 25), (3, 38)]. Data set 400 may be a data set of length 100, 1000, 10000, 20000, 50000, 100000, 500000 or longer. A moving condition may be used to appropriately perform streaming computations on data based on the window size. The moving condition is a Boolean value of (window pointer < current timestamp - window size). Once the moving condition is "true", the window pointer is moved and the rolling sum is updated accordingly. The summation operator is initialized with a sum value of 0.

[0046] At the first timestamp, the window pointer 402 of the 20 second time window 403 points to the timestamp 10. The sum operator updates the sum value to be equal to 0+2=2, as shown in the aggregate feature calculation 404A. The move condition is then checked to determine whether the timestamp that the window pointer currently points to (e.g., the first timestamp of 10) is within the window pointer time (e.g., current timestamp-window time). The move condition equals (10<10-20) = "false", so no further action is taken.

[0047] At the second timestamp, the sum operator updates the sum value to be equal to 2+4=6, as shown in aggregate feature calculation 404A. The move condition is then checked, and is currently equal to (10<20-20) = "false", so no further action is taken.

[0048] At the third timestamp, the sum operator updates the sum value to be equal to 6+3=9, as shown in aggregate feature calculation 404A. The move condition is then checked, and is currently equal to (10<25-20) = "false", so no further action is taken.

[0049] At the fourth timestamp, the summation operator updates the sum to be equal to 9+3=12. The move condition is then checked, and the move condition is currently equal to (10<38-20) = "true", so the window pointer 402 of the 20 second time window 405 moves forward by the step size and will now point to the second timestamp, as shown in FIG. Figure 4B As shown. The summation operator then removes the value 2, which corresponds to the block through which the window pointer moves (e.g., the first timestamp of 10 seconds), because it is not in the current window span. As a result, the summation operator updates the sum value to be equal to 12-2=10, as shown in the aggregate feature calculation 404B. Additionally, because the window pointer now points to the second timestamp, the new moving condition is now equal to (20<current timestamp-20). This incremental calculation method makes the computational complexity of the calculation independent of the window size because when the window pointer 402 is moved to the position of the 20-second time window 405, the rolling sum is modified by subtracting the value leaving the current window span and adding the value entering the current window span.

[0050] C. Reuse of values ​​in windows of different sizes

[0051] As described above in Section II.A, multiple windows can be applied to a single dataset without copying or reproducing the data. By using window pointers, time windows of different sizes can be applied to a single dataset to compute different aggregate features of the dataset.

[0052] Figure 5 4 shows a block diagram of two and aggregate feature calculations performed using a 20 second time window 503 and a 30 second time window 506 according to an embodiment. To calculate different aggregate features of the data set 400, multiple windows may be applied to Figure 4A and Figure 4B The same data set 400. The 20 second time window 503 can calculate the sum of the interaction values ​​in the last 20 seconds, as described above in Figure 4A and Figure 4B The 30 second time window 506 can be obtained by using a method similar to Figure 4A and Figure 4B The process described in the above slides the 30-second time window pointer 408 through the data set 400 to simultaneously or subsequently calculate the sum of the interaction values ​​in the last 30 seconds. The movement condition of the 30-second window pointer 508 for the 30-second time window will be (30-second window pointer < current timestamp - 30 seconds). This movement condition means that the 30-second window pointer 508 stays at the first timestamp that is 10 seconds longer than the 20-second window pointer 502.

[0053] Scalable window aggregators allow efficient summation of interaction values ​​over time windows. Many current implementations of aggregate feature computations store data associated with a window. Figure 5 As an example, a 20 second time window 503 will copy the data set 400 between at least the 10 second timestamp and the 25 second timestamp. When the 20 second window pointer 502 of the 20 second time window 503 slides over the data set 400, or when other windows such as the 30 second time window 506 are applied to the data set 400, many copies of the data set 400 will be created. For example, a first copy of the data set 400 will be created for the 20 second time window 503, and a second copy of the data set 400 will be created for the 30 second time window 506. Additional copies of the data are required to calculate the sum of the interaction values ​​of different time windows. One can see that if both the 20 second time window 503 and the 30 second time window 506 slide over the data set 400, many copies of the data set 400 will be needed. The embodiment instead simply uses the window pointer of the time window as a pointer to the node of the data set 400, so that the data set 400 does not need to be reproduced, which significantly reduces the memory requirements for calculating aggregate features.

[0054] Similar to the calculation of the sum of the interaction values ​​in the last 20 seconds, the aggregate feature of the count of interactions in the last 20 seconds can be calculated in a similar manner. For example, if the value column is labeled "Count" instead of "Value", the aggregate feature of the count of interactions in the last 20 seconds would be calculated instead. Further, if the sum of the interaction values ​​in the last 20 seconds and the count of interactions in the last 20 seconds are calculated (simultaneously or otherwise), the average interaction value in the last 20 seconds can be easily calculated by dividing the sum of the interaction values ​​in the last 20 seconds by the count of interactions in the last 20 seconds. More specifically, the sum operator can be applied to the data set 400, such as Figure 4A and Figure 4B The average operator may be applied to the data set 400 to determine the sum of the interaction values ​​in the last 20 seconds as described in , and the count operator may be applied to the data set 400 to determine the count of interactions in the last 20 seconds in a similar manner. The average operator may then be applied to the data set 400 and the sum of the interaction values ​​in the last 20 seconds calculated by the sum operator may be retrieved, and then the count of interactions in the last 20 seconds may be retrieved. The average operator may then determine the average interaction value in the last 20 seconds by dividing the sum of the interaction values ​​in the last 20 seconds by the count of interactions in the last 20 seconds.

[0055] D. Maximum and minimum aggregate feature calculations

[0056] The scalable window aggregator framework can be used to compute other types of aggregate features. Other examples of aggregate features include the maximum value, minimum value, or unique count of a dataset.

[0057] Fig. 6A and Figure 6B A block diagram of minimum and maximum aggregate feature calculations according to an embodiment is shown. Figure 5 The data set 600 used in the example shown in can include at least the (value, timestamp) combinations of [(2, 10), (4, 20), (3, 25), (5, 38), (8, 41), (28, 49)]. The data set 600 can have any length, such as 100, 1000, 10000, 20000, 50000, 100000, 500000 or longer. A 30-second time window 603 can be applied to the data set 600. Similar to the process described with reference to FIG4, a window pointer 602 of the 30-second time window 603 can slide over the data set 600. A maximum operator or a minimum operator can use a tree graph such as a red-black tree graph to calculate the maximum and / or minimum aggregate features of the data set.

[0058] exist Fig. 6A and Figure 6B In the example, a red-black tree map is used. The red-black tree map orders nodes (e.g., values) according to at least the following properties: (1) it obeys the binary search tree properties (e.g., if the value of the child node is less than the value of the parent node, then the child node is to the left of the parent node, and if the value of the child node is greater than the value of the parent node, then the child node is to the right of the parent node), (2) every node is either black or red, (3) the root node is black, (4) the children of a red node must be black, and (5) every path from a parent node to an empty node contains the same number of black nodes.

[0059] At the first timestamp, the window pointer 602 of the 30 second time window 603 points to timestamp 10. The maximum operator updates the tree map 610A to include "2", such as Fig. 6A As shown. Then the move condition is checked to determine whether the timestamp currently pointed to by the window pointer 602 (e.g., the first timestamp of 10) is within the window pointer time (e.g., current timestamp-window size). The move condition is equal to (10<10-30) = "false", so no further action is taken.

[0060] At the second timestamp, the maximum operator updates the tree graph 610A to include "4", as Fig. 6A As shown. Sorting of the treemap 610A is then performed according to the red-black treemap properties described above. The move condition is then checked, and the move condition is currently equal to (10<20-30) = "false", so no further action is taken.

[0061] At the third time stamp, the maximum operator updates the tree graph 610A to include "3", as Fig. 6A As shown. Sorting of the treemap 610A is then performed according to the red-black treemap properties described above. The move condition is then checked, and the move condition is currently equal to (10<25-30) = "false", so no further action is taken.

[0062] At the fourth time stamp, the maximum operator updates the tree map 610A to include "5", as shown in FIG. Fig. 6A As shown. The sorting of the tree graph 610A is then performed according to the red-black tree graph properties described above. The move condition is then checked, and the move condition is currently equal to (10<38-30) = "false", so no further action is taken.

[0063] At the fifth time stamp, the maximum operator updates the tree graph 610B to include "8", as Figure 6B As shown. Then the sorting of the tree graph 610B is performed according to the red-black tree graph properties described above. Then the moving condition is checked, and the moving condition is currently equal to (10<41-30) = "true", so the window pointer 602 of the 30 second time window 603 moves forward by the step size and will now point to the second timestamp, as shown. Figure 6B As shown. The maximum operator can then determine that the maximum value of the previous window is equal to the value "5". The maximum operator then removes the value 2, which corresponds to the block through which the window pointer moves (e.g., the first timestamp of 10 seconds), because it is not in the current window span. Since the window pointer 602 now points to the second timestamp, the new move condition is now equal to (20 < current timestamp - 30).

[0064] At the sixth time stamp, the maximum operator updates the tree graph 610B to include "28", as Figure 6B As shown. The sorting of the tree graph 610B is then performed according to the red-black tree graph properties described above. The move condition is then checked, and the move condition is currently equal to (20<49-30) = "false", so no further action is taken. The maximum operator can later determine that the maximum value of the current window is equal to "28".

[0065] In some implementations, the maximum and minimum values ​​may be determined simultaneously by applying a minimum operator as well as a maximum operator. For example, when the move condition is "true" at the fifth timestamp, the minimum operator may retrieve the tree map 610B generated by the maximum operator. The minimum operator may then be applied to the tree map 610B to determine that the minimum value of the previous window is equal to the value "2". Similarly, after the sixth timestamp, the minimum operator may determine that the minimum value of the window is equal to "3". By applying the minimum operator and the maximum operator simultaneously, the memory requirements for storing the tree map and the computational complexity of maintaining the tree map are reduced because they share the tree map with each other.

[0066] like Fig. 6A and Figure 6B As shown, a maximum operator or a minimum operator can use a tree graph to calculate the maximum or minimum value of a data set. A unique count operator can use a tree graph calculated by a maximum operator. For example, the size of the tree graph can be used to determine the unique count of the interaction value. Fig. 6AThe treemap 610A in has a size of 4, which corresponds to the unique counts of interaction values ​​that occurred (e.g., 2, 3, 4, and 5 occurred in the first window span), and Figure 6B The treemap 610B has a size of 5 (eg, 3, 4, 5, 8, and 28 appear in the second window span).

[0067] III. Computational Structure

[0068] Figure 7 A block diagram of a system architecture according to an embodiment is shown. The system architecture can be used to calculate aggregate features of a data set. The system architecture can be modified to accommodate the calculation of desired aggregate features of a data set. For example, although Figure 7 Three windows are shown, but more or fewer windows may be used in the system architecture to compute aggregate features of the data set.

[0069] A. Operator

[0070] The system architecture may include a plurality of operators, such as a first operator 700, a second operator 702, a third operator 704, a fourth operator 706, and a fifth operator 708. For example, the first operator 700 may be a sum operator, the second operator 702 may be a counting operator, the third operator 704 may be a maximum operator, the fourth operator 706 may be a minimum operator, and the fifth operator 708 may be a distinct counting operator. In some examples, each operator may be constructed such that they operate on primitive data types (e.g., integers, floating points, strings, etc.) to further reduce memory requirements and improve computational efficiency at the expense of code complexity.

[0071] B. Window

[0072] The system architecture may also include multiple windows, such as a first window 710, a second window 712, and a third window 714. The multiple windows may correspond to time intervals for which aggregate features are to be calculated. For example, if the sum of the interaction values ​​in the last 10 minutes, the count of approved interactions in the last 30 minutes, and the maximum value of the interaction values ​​in the last 4 hours are desired, the first window 710 may be a 10-minute window, the second window 712 may be a 30-minute window, and the third window 714 may be a 4-hour window.

[0073] C. Data Field

[0074] The system architecture may additionally include a plurality of data fields of the input data set, such as an order by data field 716, a first data field 718, and a second data field 720. The order by data field 716 may correspond to a timestamp of the data set. For example, the timestamp field of the data set 100 may correspond to the order by data field 716. The first data field 718 and the second data field 720 may correspond to other data fields of the input data set. For example, the first data field 718 may correspond to a value field of the interaction data set 200, and the second data field 720 may correspond to an approval count field of the interaction data set 200.

[0075] D. Computation sharing

[0076] like Figure 7 As shown, some operators can share windows, and some windows can share data fields. For example, the minimum interaction value in the last 30 minutes and the maximum interaction value in the last 30 minutes can be similarly calculated. In order to calculate the described aggregate features, a third operator 704 (e.g., a minimum operator) with a second window 712 (e.g., a 30-minute window) can be applied to a first data field 718 (e.g., an interaction value data field), and a fourth operator 706 (e.g., a maximum operator) with a second window 712 can be applied to the first data field 718. In some examples, different operators can share calculations with each other. As a simple example, an average operator can retrieve the results of a sum operator and a count operator, and calculate the average as average = sum / count. Another example can include a minimum operator retrieving a tree map generated by a maximum operator to determine a minimum value in addition to a maximum value (or vice versa). Yet another example can include a unique count operator retrieving a tree map created by a maximum operator (or a minimum operator), and calculating a unique count as the size of the tree map.

[0077] IV. Methods

[0078] Figure 8 A flow chart of a method 800 according to an embodiment is shown. The method 800 may apply a scalable window aggregator to a data set to calculate aggregate features of the data set. The method 800 may be performed by a computer, such as a server computer in communication with various external devices.

[0079] At step S808, the computer may apply the windowed operator to the processed data set to calculate aggregate features of the data set. At each step of the calculation, a move condition may be checked to determine whether the current window pointer of the window should be moved. If the move condition is met, the window pointer may be moved, and values ​​outside the window may be removed from the calculation. For example, Figure 4A and Figure 4BAs described above, after reaching the fourth timestamp, the moving condition is satisfied, and thus the window pointer moves forward. As a result of the window pointer moving forward, the old values ​​corresponding to timestamps outside the current window are removed from the calculation of and. In some examples, if multiple aggregate features are to be calculated, the computer may retrieve the first aggregate feature calculated by the first operator to calculate the second aggregate feature. For example, Fig. 6A and Figure 6B As described in , if you want to calculate the maximum value and the difference count, the difference count operator can retrieve the tree map to determine the difference count of the data set after applying the maximum operator to the data set. A similar process can occur if the sum operator and the count operator are used to calculate the sum and count of a certain window, and the average value of the same window can be easily calculated as average value = (sum / count).

[0080] At step 802, the computer may store a data set including N data measurement vectors. Each data measurement vector may include a time field and a data field, the data field including a measurement value of a property of the network system corresponding to the time value. For example, the computer may obtain the data set from an external computer such as an interaction processing computer, a network flow computer, etc. Exemplary data fields include values, identifiers, approved counts, rejected counts, etc. The data set may include any number of N data measurement vectors, such as 100, 1000, 10000, 20000, 50000, 100000, 500000 or longer. An example of a data set includes Figure 2 In some examples, the computer may additionally process the data set. In some examples, the computer may process the data set by inputting the data set into a distributed system so that parts of the data set are mapped to different processing nodes. An example of such a distributed system includes the SPARK framework. In other examples, the computer may process null values ​​appearing in the data set by appending identifiable random values ​​to the null values ​​so that the null values ​​are evenly mapped to all processing nodes. In such examples, the operator may be configured to skip null values ​​in the calculation of the aggregated feature.

[0081] In step 804, the computer may receive a command for calculating an aggregate feature vector of aggregate features of a data set. The aggregate feature vector may include corresponding aggregate values ​​of data fields of each sliding time window in a plurality of sliding time windows across the data set. For example, the computer may receive a command for calculating the sum of interaction values ​​in the last 20 seconds. In this example, the aggregate feature vector will include the sum of interaction values ​​for each 20-second time window of the data set. In some examples, the computer may receive a command for calculating more than one aggregate feature vector of the data set. For example, the computer may receive a command for calculating the sum of interaction values ​​of the data set in the last 20 seconds, the minimum value of the data set, the maximum value of the data set, and the difference count of the data set.

[0082] In step 806, after receiving a command for calculating an aggregate feature vector of aggregate features of a data set, the computer may determine an operator for determining a corresponding aggregate value and a window size for the aggregate feature to be calculated. Examples of operators may include a sum operator, a counting operator, an average operator, a maximum operator, a minimum operator, a different count operator, and the like. In some examples, operators may be constructed so that they operate on primitive data types to reduce memory requirements and improve computational efficiency of aggregate feature calculations. Examples of window sizes may include 1 second, 10 seconds, 20 seconds, 30 seconds, 10 minutes, 1 hour, 1 day, 1 year, and the like. For example, a computer may receive a command for calculating the sum of interaction values ​​of a data set in the last 20 seconds. The computer may identify the "sum" to be calculated for the "interaction value" of the data set using a time window of "20 seconds". The computer may then determine a sum operator and a 20 second time window is required to calculate the desired aggregate feature. In some examples, the computer may additionally determine moving conditions (e.g., such as Figure 4A / Figure 4B and Fig. 6A / Figure 6B An example of a moving condition is (window pointer < current timestamp - window size).

[0083] At step 808, after determining the operator and window size for the aggregate feature to be calculated, for the first time window, the computer may determine a first aggregate value using the measurement values ​​of the first data measurement vector having the time field within the first time window. Figure 4A As shown, the window pointer can be used to define a first time window (e.g., the window pointer 402 initially points to timestamp 10, so that the first time window is 10 to 30 seconds). Then, a first aggregate value can be calculated for the first time window. Figure 4A , the sum of the interaction values ​​in the last 20 seconds can be determined for a timestamp of 10. Figure 4AAs shown, the resulting sum can be calculated to be equal to "2", and a similar process is performed for each timestamp in the first time window to determine each sum in the aggregate feature calculation 404A. Fig. 6A In another example shown, a first aggregate value may be determined from the measured values ​​of the data set 600 to the tree map 610A. A maximum value in the last 30 seconds may then be determined from the tree map 610A.

[0084] At step 810, after determining the first aggregate value, the computer may advance the window pointer to define a second time window that overlaps the first time window. The second time window may include a second data measurement vector from the data set, and one or more of the second data measurement vectors may be the same as one or more of the first data measurement vectors. As a result of satisfying the move condition, the computer may advance the window pointer. For example, once the sum of the timestamps of 38 is determined, then Figure 4A The moving condition can then cause the window pointer 402 to point to a timestamp of 20, such as Figure 4B shown. Figure 4A 20 second time window in 403 and Figure 4B The 20 second time windows 403 in overlap because they both contain timestamps (20, 25, and 38). Fig. 6A and Figure 6B A similar example is described, where the window pointer 602 is moved after reaching a timestamp of 41, and Fig. 6A The 30-second time window 603 and Figure 6B The 30-second time windows 603 share common elements (20, 25, and 38).

[0085] At step 812, after advancing the window pointer, the computer may identify a first portion of the first data measurement vector that is not within the second time window. Figure 4A and 4B , after the window pointer 402 is moved to point to the timestamp 20 and the second time window is defined (e.g., 20 to 40 seconds), the computer can determine that the timestamp 10 is not in the second time window. Fig. 6A and Figure 6B Another example is shown where after window pointer 602 is moved to point to timestamp 20, the computer may determine that timestamp 10 is not in the second time window.

[0086] At step 814, after advancing the window pointer, the computer may identify a second portion of the second data measurement vector that is not within the first time window. Figure 4A and Figure 4B, after window pointer 402 is moved to point to timestamp 20, the computer may determine that timestamp 38 is included in the second time window of 20 to 40 seconds, but not in the first time window of 10 to 30 seconds. The computer may then represent the value associated with the identified timestamp (e.g., value 28 associated with timestamp 38). Similarly, in Fig. 6A and Figure 6B , after window pointer 602 is moved to point to timestamp 20, the computer can determine that timestamps 41 and 49 are included in the second time window of 20 to 50 seconds, but not in the first time window of 10 to 40 seconds. The computer can then identify the values ​​associated with the identified timestamps (e.g., a value of 8 associated with timestamp 41 and a value of 28 associated with timestamp 49).

[0087] At step 816, after identifying the first portion of the first data measurement vector that is not within the second time window and the second portion of the second data measurement vector that is not within the first time window, the computer may determine a second aggregate value using an operator. The second aggregate value may be determined by removing the measurement value of the first portion of the first data measurement vector from the first aggregate value and including the measurement value of the second portion of the second data measurement vector in the first aggregate value. For example, referring to Figure 4B , after moving the window pointer 402, the sum of timestamp 38 is calculated by adding the value of the non-overlapping portion of the second time window (3) to the previous sum (9) and subtracting the value of the non-overlapping portion of the first time window (2) to calculate a sum of 12. Similarly, referring to Figure 6B , after the window pointer 602 is moved, the tree map 610B is modified to include the values ​​of the non-overlapping portion of the second time window (8, 28), and the value of the non-overlapping portion of the first time window (2) is removed to determine the maximum value from the tree map 610B.

[0088] At step 818, the computer may repeat the advancement of the window pointer for consecutive time windows and repeat the identification of data measurement vectors for removal and inclusion so that the operator determines consecutive aggregate values ​​for consecutive time windows in the plurality of sliding time windows.

[0089] Embodiments of the present invention have many benefits. Embodiments provide a scalable window aggregator for computing aggregate features of a data set. Embodiments receive an indication of an aggregate feature to be computed and determine an appropriate operator and window. Embodiments provide a reduction in memory requirements and an increase in computational efficiency for computing aggregate features of large data sets. For example, by sharing windows by operators and data fields by windows, a reduction in memory requirements is provided because data fields do not need to be reproduced for different windows. Additionally, embodiments provide a method for reducing skew caused by null values ​​and improving computational efficiency. Null values ​​in a data set can start with identifiable random values ​​so that they are evenly distributed across the processing nodes of a computer.

[0090] V.Computer System

[0091] Fig. 9 A block diagram of an exemplary computer system 10 is shown in accordance with an embodiment of the present invention.

[0092] Any computer system mentioned herein may utilize any suitable number of subsystems. Fig. 9 An example of such a subsystem is shown in the computer device 10 in FIG. In some examples, the computer system includes a single computer device, where the subsystem can be a component of the computer device. In other embodiments, the computer system can include multiple computer devices with internal components, each of which is a subsystem. The computer system can include desktop and laptop computers, tablet computers, mobile phones, and other mobile devices.

[0093] Fig. 9 The subsystems shown in FIG. 1 are interconnected via a system bus 75. Additional subsystems such as a printer 74, a keyboard 78, a storage device 79, a monitor 76 coupled to a display adapter 82, and the like are shown. Peripheral devices and I / O devices coupled to an input / output (I / O) controller 71 may be connected to the computer system through various means known in the art, such as an input / output (I / O) port 77 (e.g., USB, ). For example, I / O port 77 or external interface 81 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 10 to a wide area network (e.g., the Internet), a mouse input device, or a scanner. Interconnection via system bus 75 allows central processor 73 to communicate with each subsystem and control the execution of multiple instructions from system memory 72 or storage device 79 (e.g., a fixed disk, such as a hard drive or optical disk), as well as the exchange of information between subsystems. System memory 72 and / or storage device 79 may contain computer readable media. Another subsystem is a data collection device 85, such as a camera, microphone, accelerometer, etc. Any data mentioned herein may be output from one component to another and may be output to a user.

[0094] The computer system may include multiple identical components or subsystems, which are connected together, for example, through an external interface 81, through an internal interface or via a removable storage device, which can be connected and removed from one component to another component. In some examples, the computer system, subsystem or device can communicate through a network. In such cases, one computer can be considered a client and another computer can be considered a server, where each computer can be part of the same computer system. The client and the server can each include multiple systems, subsystems or components.

[0095] Various aspects of the embodiments may be implemented using hardware circuits (e.g., application specific integrated circuits or field programmable gate arrays) and / or in a modular or integrated manner using computer software in the form of control logic with the aid of a substantially programmable processor. As used herein, a processor may include a single-core processor, a multi-core processor on the same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will know and appreciate other ways and / or methods of implementing embodiments of the present invention using hardware and combinations of hardware and software.

[0096] Any method described herein may be performed in whole or in part by a computer system, and the computer system includes one or more processors that can be configured to perform the steps. Therefore, embodiments may relate to a computer system configured to perform the steps of any method described herein, and may have different components that perform the corresponding steps or corresponding step groups. Although presented as numbered steps, the steps of the method herein may be performed simultaneously or at different times or in different orders. For example, a data set may be processed before receiving an instruction for calculating aggregate features. In addition, parts of these steps may be used together with parts of other steps of other methods. In addition, all or part of the steps may be optional. In addition, any step of any method may be performed using other devices of the module, unit, circuit or system for performing these steps.

[0097] The specific details of a particular embodiment may be combined in any suitable manner without departing from the spirit and scope of the embodiments of the present disclosure. However, other embodiments of the present disclosure may be directed to specific embodiments related to each individual aspect, or specific combinations of these individual aspects.

[0098] The above description of example embodiments of the present disclosure has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the above teachings.

[0099] Unless expressly indicated to the contrary, the recitation of "a", "an" or "the" is intended to mean "one or more". Unless expressly indicated to the contrary, the use of "or" is intended to mean an "inclusive or" rather than an "exclusive or". Reference to a "first" component does not necessarily require the provision of a second component. Furthermore, reference to a "first" or "second" component does not limit the referenced components to a particular location unless expressly stated. The term "based on" is intended to mean "based, at least in part, on".

[0100] All patents, patent applications, publications and descriptions mentioned in this article and the appendix are incorporated herein by reference in their entirety for all purposes. It is not admitted that they are prior art. When there is a conflict between the present application and the references provided herein, the present application shall prevail.

Claims

1. A method for processing a data set by a computer, the method comprising: storing the data set comprising N data measurement vectors, each data measurement vector comprising a time field and a data field, wherein N is at least 100,000, wherein the data field of each data measurement vector comprises a measurement value of a property of the network system corresponding to the time value; receiving a command for computing an aggregate feature vector of aggregate features of the data set, wherein the aggregate feature vector comprises respective aggregate values ​​of the data field across each of a plurality of sliding time windows of the data set; determining an operator for determining a corresponding aggregate value and a window size for the aggregate feature to be calculated; for a first time window, determining a first aggregate value using the operator using measurement values ​​of a first data measurement vector having the time field within the first time window; advancing a window pointer to define a second time window, wherein the second time window overlaps the first time window, and wherein the second time window includes a second data measurement vector; identifying a first portion of the first data measurement vector that is not within the second time window; identifying a second portion of the second data measurement vector that is not within the first time window; The second aggregate value is determined using the operator in the following manner: removing the measurement value(s) of the first portion of the first data measurement vector from the first aggregate value; and including the measurement value(s) of the second portion of the second data measurement vector in the first aggregate value; as well as The advancing of the window pointer is repeated for successive time windows, and the identification of data measurement vectors for removal and inclusion is repeated, such that the operator determines successive aggregate values ​​for the successive time windows in the plurality of sliding time windows.

2. The method of claim 1, wherein the data set comprises interaction data, the interaction data comprising one or more of a value, an approved interaction count, a rejected interaction count, or an identifier.

3. The method of claim 1, further comprising: The data set is processed using a distributed system to distribute portions of the data set to different processing nodes of the computer.

4. The method of claim 3, wherein processing the data set comprises modifying null values ​​in the data set to include identifiable random values, and wherein the operator is configured to skip the null values ​​in the calculation of the aggregate feature of the data set. 5 . The method of claim 1 , wherein the aggregate feature of the data set comprises one of a value, a count, an average, a maximum value, a minimum value, or a distinct count. The method of claim 1 , wherein the window pointer is advanced after a move condition is satisfied.

7. The method of claim 6, wherein the movement condition is defined by at least a value in the time field, the window size, and the window pointer.

8. The method of claim 1, wherein the aggregate feature vector is a first aggregate feature vector of a first aggregate feature, the method further comprising: receiving a second command for calculating a second aggregate feature vector for a second aggregate feature of the data set, wherein the second aggregate feature is different from the first aggregate feature of the data set; Determining a second window size of the second aggregate feature to be calculated; as well as The second aggregate feature is calculated using the operator, the second window size, and the measurement values ​​of the first data measurement vector.

9. The method of claim 1, wherein the processed data set comprises an expanded linked list.

10. The method of claim 1, wherein the aggregate feature vector is a first aggregate feature vector of a first aggregate feature, the method further comprising: receiving an instruction for calculating a second aggregated feature vector of second aggregated features of the data set; retrieving the first aggregated feature vector of the data set; as well as The second aggregated feature vector of the data set is calculated using at least the first aggregated feature vector of the data set.

11. The method of claim 10, wherein the first aggregate feature is a sum or count and the second aggregate feature is an average, or wherein the first aggregate feature is a maximum or minimum value and the second aggregate feature is a distinct count.

12. The method of claim 1, wherein the operator is one of a sum operator, a count operator, an average operator, a maximum operator, a minimum operator, or a distinct count operator.

13. The method of claim 1, wherein determining the first aggregate value using the operator using the measurement values ​​of the first data measurement vector having the time field within the first time window includes constructing a tree graph using the measurement values ​​of the first data measurement vector.

14. The method of claim 1, wherein the aggregate feature comprises an indication of an operation and a time interval, and wherein the operator is determined based on the operation and the sliding time window is determined based on the time interval.

15. A computer, comprising: processor; as well as A non-transitory computer-readable medium comprising instructions executable by the processor to perform the method of any one of claims 1 to 14.

Images