Data processing method, stream processing system, medium and device based on sliding window

Abstract

The present disclosure provides a data processing method based on a sliding window, a stream processing system, a data processing device based on a sliding window, a computer-readable storage medium and an electronic device, which belong to the field of data processing technology. The method is applied to a stream processing system, and includes: determining the target sliding window of the received stream data and the shard position of the stream data in the target sliding window according to a preset division rule; calling an aggregation algorithm to aggregate the stream data at each shard position in the target sliding window to obtain an aggregation result; using the aggregation result to update a tree structure constructed using a circular array, wherein the leaf nodes and non-leaf nodes of the tree structure are respectively used to store the stream data in the target sliding window and the local aggregation results of the stream data in the target sliding window; and determining the calculation results of the stream data in the target sliding window through the nodes of the tree structure. The present disclosure can improve the real-time processing and processing efficiency of stream data and reduce the resource overhead of stream data processing.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method based on a sliding window, a stream processing system, a data processing device based on a sliding window, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the development of the Internet and the widespread use of mobile terminal applications, huge amounts of data have been generated. In order to provide users with better services and optimize the service structure, it is often necessary to analyze and process this data.

[0003] In data-intensive applications, streaming data is often generated rapidly and ephemeral, requiring timely data processing. For example, applications such as network monitoring, telecommunications data management, sensor networks, and quantitative trading require processing of real-time streaming data. Existing methods can process unbounded streaming data in batch mode, but this approach cannot meet the real-time requirements of streaming data. Therefore, there is a need for a method that can improve the real-time performance of streaming data processing.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The present disclosure provides a data processing method based on a sliding window, a stream processing system, a data processing device based on a sliding window, a computer-readable storage medium and an electronic device, thereby at least to a certain extent improving the problem of low real-time performance of stream data processing in the prior art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a data processing method based on a sliding window is provided, which is applied to a stream processing system, and the method includes: determining a target sliding window of received stream data and a shard position of the stream data within the target sliding window according to a preset division rule; calling an aggregation algorithm to perform aggregation processing on the stream data at each shard position within the target sliding window to obtain an aggregation result; using the aggregation result to update a tree structure constructed using a circular array, wherein the leaf nodes and non-leaf nodes of the tree structure are respectively used to store the stream data within the target sliding window and the local aggregation results of the stream data within the target sliding window; and determining the calculation results of the stream data within the target sliding window through the nodes of the tree structure.

[0008] In an exemplary embodiment of the present disclosure, the method further includes: determining whether the reception time of the stream data is greater than or equal to a dynamic arrival time, the dynamic arrival time being determined based on the time difference between the maximum event time of the stream data entering the window and a preset delay time; when the reception time of the stream data is greater than or equal to the dynamic arrival time, triggering determination of a target sliding window for the stream data.

[0009] In an exemplary embodiment of the present disclosure, determining a target sliding window of received stream data and a shard position of the stream data within the target sliding window according to a preset division rule includes: determining the target sliding window of the stream data and the shard position of the stream data within the target sliding window according to data parameters of the stream data and window parameters of each window; wherein the data parameters of the stream data include the receiving time and receiving order of the stream data, and the window parameters of each window include the window length and window type of each window.

[0010] In an exemplary embodiment of the present disclosure, determining the target sliding window of the stream data and the shard position of the stream data within the target sliding window according to the data parameters of the stream data and the window parameters of each window includes: when the reception time of the stream data is greater than the window length of each sliding window, creating a new sliding window and determining the new sliding window as the target sliding window of the stream data; when the reception time of the stream data is not greater than the window length of any sliding window, determining any sliding window as the target sliding window of the stream data.

[0011] In an exemplary embodiment of the present disclosure, each shard position in the target sliding window corresponds to a data shard, and the data shard is composed of one or more stream data.

[0012] In an exemplary embodiment of the present disclosure, when calling an aggregation algorithm to perform aggregation processing on the streaming data at each shard position within the target sliding window, the method further includes: when the moving step size of the target sliding window is less than the window length, determining the duplicate shards within the target sliding window; calculating the aggregation results of other data shards within the target sliding window except the duplicate shards, and using the aggregation results of the duplicate shards and the aggregation results of the other data shards to determine the aggregation results of the streaming data within the target sliding window.

[0013] In an exemplary embodiment of the present disclosure, the method further includes: when aggregating the stream data at each shard position in the target sliding window, determining whether all the stream data at each shard position in the target sliding window has arrived; when all the stream data at any shard position in the target sliding window has not arrived, receiving new stream data, and adding the new stream data to any shard position according to the preset division rule; when all the stream data at each shard position in the target sliding window has arrived, adding the stream data at each shard position to the leaf node of the tree structure.

[0014] In an exemplary embodiment of the present disclosure, after aggregating the stream data at each shard position within the target sliding window, the method further includes: updating the data shards at each shard position within the target sliding window according to the moving step of the target sliding window; and calculating the aggregation results of the data shards at each shard position within the target sliding window after the data shards are updated.

[0015] In an exemplary embodiment of the present disclosure, the updating of data shards at each shard position within the target sliding window according to the step size of the target sliding window includes: deleting expired shards within the target sliding window, and deleting the node data of the expired shards from the tree structure, wherein the expired shards are data shards at the shard positions within the target sliding window where aggregation processing has been completed; receiving new stream data, and adding the new stream data to the shard positions corresponding to the expired shards within the target sliding window according to the preset division rules.

[0016] In an exemplary embodiment of the present disclosure, the shard range corresponding to the stream data in the target sliding window is located by using head and tail pointers.

[0017] According to a third aspect of the present disclosure, a stream processing system is provided, which includes: a determination node for determining a target sliding window of received stream data and a shard position of the stream data within the target sliding window according to a preset division rule; a calculation node for calling an aggregation algorithm to perform aggregation processing on the stream data at each shard position within the target sliding window to obtain an aggregation result, and using the aggregation result to update a tree structure constructed using a circular array; a storage node for storing the tree structure; wherein the leaf nodes and non-leaf nodes of the tree structure are respectively used to store the stream data within the target sliding window and the local aggregation results of the stream data within the target sliding window.

[0018] In an exemplary embodiment of the present disclosure, the system also includes a trigger node for determining whether the reception time of the stream data is greater than or equal to the dynamic arrival time, and when the reception time of the stream data is greater than or equal to the dynamic arrival time, triggering the determination of the target sliding window of the stream data; wherein the dynamic arrival time is determined based on the time difference between the maximum event time of the stream data entering the window and the preset delay time.

[0019] According to a third aspect of the present disclosure, a data processing device based on a sliding window is provided, which is applied to a stream processing system and includes: a first determination module, used to determine a target sliding window of received stream data and a shard position of the stream data within the target sliding window according to a preset division rule; an aggregation module, used to call an aggregation algorithm to perform aggregation processing on the stream data at each shard position within the target sliding window to obtain an aggregation result; an update module, used to use the aggregation result to update a tree structure constructed using a circular array, wherein the leaf nodes and non-leaf nodes of the tree structure are used to store the stream data within the target sliding window and the local aggregation results of the stream data within the target sliding window, respectively; a second determination module, used to determine the calculation results of the stream data within the target sliding window through the nodes of the tree structure.

[0020] In an exemplary embodiment of the present disclosure, the first determination module is used to determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time, and the dynamic arrival time is determined based on the time difference between the maximum event time of the stream data entering the window and the preset delay time. When the reception time of the stream data is greater than or equal to the dynamic arrival time, it triggers the determination of the target sliding window of the stream data.

[0021] In an exemplary embodiment of the present disclosure, the first determination module is used to determine the target sliding window of the stream data and the shard position of the stream data within the target sliding window based on the data parameters of the stream data and the window parameters of each window, wherein the data parameters of the stream data include the receiving time and receiving order of the stream data, and the window parameters of each window include the window length and window type of each window.

[0022] In an exemplary embodiment of the present disclosure, the first determination module is further configured to create a new sliding window when the reception time of the stream data is greater than the window length of each sliding window, and determine the new sliding window as the target sliding window of the stream data; and when the reception time of the stream data is not greater than the window length of any sliding window, determine any sliding window as the target sliding window of the stream data.

[0023] In an exemplary embodiment of the present disclosure, each shard position in the target sliding window corresponds to a data shard, and the data shard is composed of one or more stream data.

[0024] In an exemplary embodiment of the present disclosure, when calling an aggregation algorithm to perform aggregation processing on the stream data at each shard position in the target sliding window, the aggregation module is used to determine the duplicate shards in the target sliding window when the moving step size of the target sliding window is less than the window length, calculate the aggregation results of other data shards in the target sliding window except the duplicate shards, and use the aggregation results of the duplicate shards and the aggregation results of the other data shards to determine the aggregation results of the stream data in the target sliding window.

[0025] In an exemplary embodiment of the present disclosure, the aggregation module is also used to determine whether the stream data of each shard position in the target sliding window has all arrived when aggregating the stream data at each shard position in the target sliding window. When the stream data of any shard position in the target sliding window has not all arrived, new stream data is received and the new stream data is added to any shard position according to the preset division rule. When the stream data of each shard position in the target sliding window has all arrived, the stream data of each shard position is added to the leaf node of the tree structure.

[0026] In an exemplary embodiment of the present disclosure, after aggregating the stream data at each shard position within the target sliding window, the update module is used to update the data shards at each shard position within the target sliding window according to the moving step of the target sliding window, and calculate the aggregation results of the data shards at each shard position within the target sliding window after the data shards are updated.

[0027] In an exemplary embodiment of the present disclosure, the update module is also used to delete expired shards within the target sliding window and delete the node data of the expired shards from the tree structure. The expired shards are data shards of the shard positions within the target sliding window that have completed aggregation processing. The module receives new stream data and adds the new stream data to the shard positions corresponding to the expired shards within the target sliding window according to the preset division rules.

[0028] In an exemplary embodiment of the present disclosure, the shard range corresponding to the stream data in the target sliding window is located by using head and tail pointers.

[0029] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned sliding window-based data processing methods is implemented.

[0030] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned sliding window-based data processing methods by executing the executable instructions.

[0031] The present disclosure has the following beneficial effects:

[0032] In summary, according to the data processing method based on a sliding window, the stream processing system, the data processing device based on a sliding window, the computer-readable storage medium, and the electronic device in this exemplary embodiment, the target sliding window of the received stream data and the shard position of the stream data within the target sliding window can be determined according to a preset partitioning rule, and the aggregation algorithm is called to aggregate the stream data at each shard position within the target sliding window to obtain an aggregation result. The aggregation result is then used to update the tree structure constructed using a circular array, and the calculation result of the stream data within the target sliding window is determined through the nodes of the tree structure. On the one hand, by calling the aggregation algorithm to aggregate the stream data at each shard position within the target sliding window, the aggregation processing of the stream data can be implemented using a sliding window, and part of the aggregation results of the stream data can be shared between sliding windows, which can reduce the impact of redundant calculations and reduce unnecessary data copy operations; on the other hand, by using the aggregation result to update the tree structure constructed using a circular array, the characteristics of the tree structure can be used to reduce the complexity of stream data processing and improve data processing efficiency.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and it is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort.

[0035] Figure 1 A flowchart showing a method for processing stream data in the related art;

[0036] Figure 2 A flow chart showing a data processing method based on a sliding window in this exemplary embodiment is shown;

[0037] Figure 3 A sub-flowchart showing a data processing method in this exemplary embodiment;

[0038] Figure 4 A schematic diagram showing a target sliding window in this exemplary embodiment is shown;

[0039] Figure 5 A flowchart showing a method of calculating an aggregation result in this exemplary embodiment is shown;

[0040] Figure 6 A flowchart showing another data processing method based on a sliding window in this exemplary embodiment is shown;

[0041] Figure 7 A schematic structural diagram of a stream processing system according to this exemplary embodiment is shown;

[0042] Figure 8 A schematic structural diagram of another stream processing system according to this exemplary embodiment is shown;

[0043] Figure 9 A schematic diagram showing classes included in a window operator in this exemplary embodiment;

[0044] Figure 10 A structural block diagram of a data processing device based on a sliding window in this exemplary embodiment is shown;

[0045] Figure 11 A computer-readable storage medium for implementing the above method in this exemplary embodiment is shown;

[0046] Figure 12 An electronic device for implementing the above method in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In one solution of the related technology, the stream data can be converted into bounded data by using the sliding window partitioning method through the stream computing framework, and then the data in the window can be processed. Figure 1As shown, a sliding window can be used to copy data to the window to which the data belongs. For example, for a sliding window with a window size of M and a moving step of N, where M and N are both numbers greater than 0, the data needs to be copied M / N times and the repeated data needs to be calculated M / N times. When the window size is larger than the moving step, since each window cannot estimate the amount of data, it needs to frequently apply for small memory to store the incoming stream data, resulting in a slow system response. At the same time, since each window processes the stream data independently, there are a lot of overlapping parts between different windows. Independent processing between windows will cause the calculation results of the overlapping parts to be unable to be shared by other windows, resulting in excessive redundant calculations and a large amount of waste of computing resources.

[0049] In view of one or more of the aforementioned problems, the exemplary embodiments of the present disclosure first provide a data processing method based on a sliding window. This method can be applied to a stream processing system, so that it can use a sliding window to aggregate the acquired stream data and determine the calculation results of the stream data based on the results of the aggregation processing. For example, in an application with an information recommendation function, the above method can be used to count the number of users who viewed the recommended information within 10 minutes. Specifically, a sliding window, such as a time sliding window, can be defined to collect user click events within the past 10 minutes, and statistical calculations can be performed on the data within each time sliding window.

[0050] Figure 2 A process of this exemplary embodiment is shown, which may include the following steps S210 to S240:

[0051] Step S210: Determine the target sliding window of the received stream data and the slice positions of the stream data within the target sliding window according to a preset partitioning rule.

[0052] A target sliding window refers to a sliding window used to process stream data. A stream data can have one or more target sliding windows, and different target sliding windows can perform different processing on the stream data. A shard position refers to the position of a data shard consisting of stream data within a target sliding window. A target sliding window can generally include multiple shard positions, each of which corresponds to a data shard, which can be composed of one or more stream data. For example, stream data can be generated every second, and a data shard can include one stream data, i.e., stream data generated within 1 second, or it can include multiple stream data, such as stream data generated within 10 seconds. In this exemplary embodiment, the shard positions can be arranged according to the reception time of the stream data, and the size of the data shards at each shard position can be the same or different. The preset partitioning rules can be customized by the operator based on the data content and reception time of the stream data, as well as the window type and window length, etc. This exemplary embodiment does not specifically limit this.

[0053] A characteristic of streaming data is its continuous arrival. To facilitate streaming data processing, the target sliding window for each received stream data item and the shard position of the stream data within the target sliding window can be determined according to corresponding partitioning rules. For example, the target sliding window for streaming data can be determined based on the time the stream data was received, so that the stream data in the target sliding window has a continuous reception time, and the stream data at each shard position within the target sliding window is arranged in chronological order of reception time. In this method, the data in each sliding window can be composed of multiple shards, realizing the use of shards to store streaming data and minimizing memory usage.

[0054] Since the stream data entering the stream processing system often has problems such as data record disorder and delayed arrival due to factors such as delay and back pressure, in order to restore the order of the stream data, in an optional implementation method, Figure 3 As shown, the following methods can be executed:

[0055] Step S310: Determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time.

[0056] The reception time of stream data can be the time when the stream data arrives at the stream processing system; the dynamic arrival time is essentially a dynamically changing timestamp, which is determined based on the time difference between the maximum event time entering the window and the preset delay time. Specifically, the dynamic arrival time = the maximum event time entering the window - the specified delay time, where the maximum event time refers to the maximum reception time of the stream data entering the window, and the specified delay time refers to the user-configured time length for supporting the maximum delayed arrival. When the reception time of the stream data exceeds the specified delay time, it will be considered as delayed data or abnormal data.

[0057] Step S320 : When the reception time of the stream data is greater than or equal to the dynamic arrival time, triggering the determination of the target sliding window of the stream data.

[0058] When the stream data's reception time is greater than or equal to the dynamic arrival time, it indicates that all the stream data belonging to the sliding window has arrived. At this point, the target sliding window for the stream data can be determined. This method ensures the accuracy of calculation results for out-of-order data, thereby improving the accuracy of stream data processing.

[0059] Furthermore, in an optional embodiment, the target sliding window for the stream data and the slice positions of the stream data within the target sliding window may be determined based on data parameters of the stream data and window parameters of each window. The data parameters of the stream data may include the time and order of receipt of the stream data, and the window parameters of each window may include the window length and window type of each window.

[0060] For example, you can first search for a sliding window that is compatible with the stream data based on the window type and window length of each window, and then search for a window where all data has not arrived in the found sliding window based on the reception time of the stream data to determine the target sliding window of the stream data and the shard position in the target sliding window; for another example, after determining the sliding window that is compatible with the stream data, you can determine the sliding window to which the previous stream data belongs as the target sliding window of the stream data according to the reception order of the stream data, and arrange the stream data according to the reception order so that it is in the corresponding shard position.

[0061] Specifically, in an optional implementation, the following method may be performed to determine the target sliding window of the stream data:

[0062] When the receiving time of the stream data is greater than the window length of each sliding window, a new sliding window is created and the new sliding window is determined as the target sliding window of the stream data;

[0063] When the reception time of the stream data is not greater than the window length of any sliding window, the above-mentioned any sliding window is determined as the target sliding window of the stream data.

[0064] When the time it takes to receive stream data exceeds the window length of each sliding window, it indicates that the data arriving in each sliding window has reached the maximum limit of the window. In this case, the newly created sliding window can be determined as the target sliding window for the stream data. Conversely, when the time it takes to receive stream data does not exceed the window length of any sliding window, it indicates that there are still data gaps in any of the aforementioned sliding windows. In this case, any of the aforementioned sliding windows can be determined as the target sliding window for the stream data. This method allows stream data to be divided into corresponding sliding windows based on their reception time, ensuring accurate window calculation for the stream data.

[0065] Step S220: Call the aggregation algorithm to aggregate the stream data at each shard position in the target sliding window to obtain an aggregation result.

[0066] The aggregation algorithm can be implemented through the aggregation function pre-written by the operator. The aggregation algorithm can be used to count and aggregate the stream data in the window; the aggregation result refers to the result obtained after the aggregation processing of the stream data in the target sliding window, which can include the result obtained by directly aggregating the stream data, or the result generated by one or more aggregations of the aggregation result after the aggregation of the stream data.

[0067] Specifically, when performing aggregation processing, an aggregation algorithm can be called to aggregate the stream data at each shard position within the target sliding window. For example, when calculating the average value of data over a period of time, assuming that there are four shard positions within the window, an aggregation algorithm can be called to perform statistics on the stream data at two adjacent shard positions within the target sliding window, and the average value of the sum of the stream data at each of the two adjacent shard positions can be calculated. The average value of the calculated average value can then be calculated again to obtain the average value of the stream data within the target sliding window. In this way, the aggregation result of the stream data within the window can be calculated, and since the data within the window is composed of data shards at each shard position, when the window length is greater than the window moving step, the repeatedly calculated aggregation results can be shared between different windows, which can reduce the impact of redundant calculations, avoid unnecessary data copying operations, and improve computing efficiency.

[0068] In an optional implementation, the shard range corresponding to the stream data in the target sliding window can be located by using the head and tail pointers. Figure 4In the target sliding window shown, the window size is 4 seconds, the movement step is 1 second, and the movement direction is from left to right. The window contains four data slices, and the value before each data slice is the array index. For seconds 0-4, the target sliding window is window-1, and the head and tail pointers are array indexes 3 and 6, respectively. For seconds 1-5, the target sliding window is window-2, and the array indexes of the data slices in this window are 4, 5, 6, and 3, respectively. Therefore, the head and tail pointers are array indexes 4 and 3, respectively. Similarly, for seconds 2-6, the head and tail pointers of the data slices in the target sliding window are 5 and 4, respectively, and for seconds 3-7, the head and tail pointers of the data slices in the target sliding window are 6 and 5, respectively. This method effectively locates the data slice range within the target sliding window and ensures the accuracy of window calculation.

[0069] During aggregation processing, if the moving step size of the sliding window is smaller than the window length, the aggregation results of some stream data within the window will be repeatedly calculated. Therefore, in order to improve the calculation efficiency, in an optional embodiment, in step S230, the following method may also be performed:

[0070] When the moving step of the target sliding window is less than the window length, duplicate fragments within the target sliding window are determined;

[0071] Calculate the aggregation results of other data shards except duplicate shards in the target sliding window, and use the aggregation results of duplicate shards and the aggregation results of other data shards to determine the aggregation results of the stream data in the target sliding window.

[0072] Among them, the repeated fragments can be the same data fragments in the target sliding window before and after a movement. For example, Figure 4 In the target sliding window shown, the repeated slices in the target sliding windows corresponding to seconds 0-4 and seconds 1-5 are data slices at the slice positions with array subscripts 4, 5, and 6.

[0073] For repeated shards within the target sliding window, each calculation can obtain the aggregated results calculated during the historical movement process. For the remaining data shards, the aggregated results within the current target sliding window can be calculated to obtain the aggregated results of all data shards within the target sliding window. This method can share the aggregated results within the target sliding window, which not only reduces the number of times the aggregated results of the data shards are calculated, improving data processing efficiency, but also greatly reduces the occupation and consumption of computer resources caused by repeated calculations, thereby improving resource utilization.

[0074] In an optional embodiment, when aggregating the stream data at each shard position in the target sliding window, it is possible to verify whether there is a data shard at each shard position in the target sliding window. If there is no data shard at any shard position, it means that the stream data has not arrived completely, and new stream data can continue to be received and added to any of the above shard positions until the data shard at that shard position is completely written. If there is a data shard at each shard position, the aggregation of the stream data at each shard position in the target sliding window can be triggered. This method ensures that all data shards in the target sliding window are fully written each time the aggregation process is performed, thereby reducing the number of calculations and improving calculation efficiency.

[0075] Furthermore, in an optional implementation, the following method may also be performed:

[0076] When aggregating the stream data at each shard position in the target sliding window, determining whether all the stream data at each shard position in the target sliding window has arrived;

[0077] When the stream data at any shard position in the target sliding window has not arrived completely, new stream data is received and added to any of the above shard positions according to the preset partitioning rules;

[0078] When all the stream data of each shard position in the target sliding window has arrived, the stream data of each shard position is added to the leaf node of the tree structure.

[0079] Through the above method, it can be ensured that all the stream data in the target sliding window arrives, avoiding errors in the aggregation results due to the lack of some stream data, and ensuring the accuracy of the calculation results.

[0080] After aggregating the stream data at each shard position within the target sliding window, the target sliding window can be moved according to the corresponding moving step size, updating the stream data within the window, and aggregating the new stream data. Figure 5 As shown, the following methods can be executed:

[0081] Step S510: Update the data slices at each slice position in the target sliding window according to the moving step of the target sliding window.

[0082] The moving step size is the length of each sliding of the target sliding window, which can be a time length, such as 1 second, or a data length, such as 1 data slice.

[0083] The data slices controlling the target sliding window are updated according to the moving step of the target sliding window. For example, the data slices within the target sliding window 1 second ago are updated to the data slices consisting of the newly received stream data, thereby completing the update of the data slices within the target sliding window.

[0084] Specifically, in an optional implementation, step S510 may also be implemented by the following method:

[0085] Delete the expired shards in the target sliding window and delete the node data of the expired shards from the tree structure;

[0086] Receive new stream data and add the new stream data to the shard position corresponding to the expired shard in the target sliding window according to the preset partitioning rules.

[0087] Expired fragments can be data fragments at the fragment position where the aggregation process is completed in the target sliding window and discarded after moving one step. After completing the aggregation process of the stream data in the target sliding window, the expired fragments and the node data of the expired fragments in the tree structure can be deleted, and then the new stream data that continues to arrive can be added to the fragment position where the expired fragment is located in the target sliding window according to the preset division rule. For example, in Figure 4 In the target sliding window shown, the data shard that has completed aggregation processing in the target sliding window window-2 is the data with array index 3, that is, the expired shard. At the 4th second, window-1 will be triggered to perform window calculation. After the window calculation is triggered, the data within 0-1s can be directly discarded, and the newly created data shard of 4-5s will also be placed at the position with array index 3. The position pointed by the head and tail pointers will also be recorded in window-2. At the 5th second, the range of data shards included in window-2 can be determined based on the head and tail pointers and aggregation calculation can be performed. And so on.

[0088] By determining the expired shards within the target sliding window, its node data is deleted from the tree structure, so that the data of the corresponding nodes in the tree structure is released, and then new stream data is received and divided into the shard position where the expired shard is located according to the above preset division rules to complete the data update.

[0089] Step S520 , calculating the aggregation results of the data shards at each shard position in the target sliding window after the data shards are updated.

[0090] Specifically, the above aggregation algorithm can be called to aggregate the data fragments at each fragment position in the target sliding window after the data fragments are updated to obtain a new aggregation result, thereby realizing the processing of stream data based on the sliding window.

[0091] Step S230: Use the aggregation result to update the tree structure constructed using the loop array.

[0092] The leaf nodes and non-leaf nodes of the tree structure can be used to store the stream data within the target sliding window and the local aggregation results of the stream data within the target sliding window, respectively. The local aggregation results can include the results of aggregating data slices at some shard positions within the target sliding window, the results of one or more aggregation processes on the aggregation results of the data slices, etc. A circular array is a continuous block of memory data that can be used to store stream data and its aggregation results.

[0093] In this exemplary embodiment, the nodes of the tree structure can represent stream data and the aggregation results of the stream data. After obtaining the aggregation results of each data slice in the target sliding window, the aggregation results can be used to update the nodes of the tree structure. For example, the aggregation results of each data slice can be child nodes of the tree structure, and the value of the parent node of the child node can be updated according to the value of the child node until the update of the root node of the tree structure is completed.

[0094] For example, in Figure 4 In the target sliding window shown, assuming two stream data items arrive per second and the circular array size is 4, the target sliding window includes four data slices, each containing two stream data items. The circular array occupies 7 memory elements. The first three elements, those with array indices 0, 1, and 2, store the aggregation results of some of the data slices. "1" and "2" are children of "0," and the elements with array indices 3, 4, 5, and 6 can be used to store stream data. From seconds 0 to 4, four data slices consisting of eight stream data items arrive sequentially and are stored in the slice positions with array indices 3, 4, 5, and 6. At the fourth second, the circular array is full, triggering window computation and obtaining the aggregation results for slices 3 and 4, and for slices 5 and 6. This aggregation result can be used to update the aggregation result of the node with array indices 0. For example, during a summation computation, the element with array indices 1 can be directly updated with the sum of the data slices with array indices 3 and 4. At the 5th second, the target sliding window moves forward 1 second, converting from window-1 to window-2. The front data shard [2 6] is updated to the data shard [6 2] that arrived 4-5 seconds later. At this time, the data shard [7 4] with array subscript 4 is located at the head of window-2. At this time, the window aggregation calculation is performed again to obtain the aggregation results of array subscripts 4 and 5 and data subscripts 6 and 3. The aggregation results are used to update the values ​​of array subscripts 0, 1, and 2 to obtain the aggregation results of the stream data of the entire target sliding window within 1-5 seconds.

[0095] In fact, when updating the tree structure, the non-leaf nodes of the tree structure before and after the target sliding window moves can save the aggregation results before the movement. On this basis, the update of the aggregation results of the stream data after the target sliding window moves in the tree structure can be based on the values ​​of the non-leaf nodes before the movement. In this way, when the target sliding window moves to the last time, the tree structure can be used to obtain the aggregation results of all the stream data passing through the target sliding window.

[0096] Through the above method, a circular array can be used to store the underlying stream data, enabling the sharing of data shards by sliding windows, thus avoiding frequent requests for small amounts of memory that reduce memory allocation efficiency. At the same time, the characteristics of the tree structure make data calculations hierarchical, reducing the complexity of the calculations and improving the calculation efficiency by reusing the data of some nodes.

[0097] Step S240: Determine the calculation result of the stream data in the target sliding window through the nodes of the tree structure.

[0098] Specifically, the leaf nodes of the tree structure can be used to determine the stream data within the target sliding window, and the non-leaf nodes of the tree structure can be used to determine the aggregation results of the stream data within the target sliding window, wherein the root node among the non-leaf nodes can represent the final aggregation result of the stream data, and the child nodes of the root node can represent the local aggregation results of the stream data.

[0099] During the stream data processing process, since the nodes in the tree structure can also store the aggregation results of the stream data of the target sliding window during the historical movement process, the nodes in the tree structure can save the aggregation results obtained by the previous movement calculation during each calculation. Therefore, after the last movement, the aggregation results of the stream data in the target sliding window can be calculated and the node data of the tree structure can be updated. The node data can be used to obtain the aggregation results of all stream data processed by the target sliding window.

[0100] Figure 6 Another data processing method based on a sliding window is shown, which may include the following steps S601 to S611:

[0101] Step S601: Receive streaming data.

[0102] Step S602: determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time.

[0103] When the receiving time of the stream data is greater than or equal to the dynamic arrival time, the target sliding window for the stream data is triggered and step S603 is executed. Conversely, when the receiving time of the stream data is less than the dynamic arrival time, it means that the arrived stream data has not reached the corresponding amount, and step S604 is executed to determine whether the receiving time of the stream data is greater than the window length of the sliding window.

[0104] Step S603: Determine the target sliding window of the stream data.

[0105] Specifically, the target sliding window of the stream data can be determined according to a preset division rule. For example, the stream data can be divided into sliding windows to which the stream data has not yet arrived, ie, target sliding windows, according to the window type of each window.

[0106] Step S604: determine whether the reception time of the stream data is greater than the window length of the sliding window.

[0107] When the reception time of the stream data is greater than the window length of the sliding window, it means that the stream data in the sliding window has reached the maximum limit, and step S605 can be executed to create a new sliding window as the target sliding window. Conversely, when the reception time of the stream data is not greater than the window length of the sliding window, it means that the stream data in the sliding window is not yet full, and step S606 is executed to determine the sliding window as the target sliding window, and add the stream data to the corresponding shard position in the target sliding window.

[0108] Step S605: Create a new sliding window as the target sliding window and create a new shard position.

[0109] The new shard position may be the position of the data shard in the target sliding window, which may store each data shard through a circular array.

[0110] Step S606: The sliding window is determined as the target sliding window, and the stream data is added to the corresponding slice position in the target sliding window. For example, the stream data can be added to each slice position in the target sliding window in sequence according to the reception time of the stream data.

[0111] Step S607: Update the partial aggregation results within the target sliding window.

[0112] After the stream data is added to the corresponding shard position in the target sliding window, the data is updated, so the corresponding aggregation algorithm can be called to aggregate the stream data in the target sliding window to obtain the aggregation result of the stream data in the target sliding window.

[0113] Step S608: Determine whether there is a data fragment at each fragment position in the target sliding window.

[0114] When there is no data shard at any shard position, step S601 can be executed to receive new stream data and add the new stream data to any of the above shard positions; when there is data shard at each shard position, step S609 can be executed to add the new data shard to the tree structure, and then step S601 can be continued to receive new stream data until the target sliding window for determining the stream data is triggered.

[0115] Step S609: Add a new data slice to the tree structure. For example, the new data slice can be added to the corresponding leaf node of the tree structure, and the aggregation result of the new data slice can be added to the non-leaf node of the tree structure.

[0116] Step S610: Calculate the aggregation results of the data shards at each shard position in the target sliding window.

[0117] Step S611: Delete the expired fragments and delete the node data of the expired fragments from the tree structure.

[0118] After the calculation is complete, the discarded expired shards and their node data can be deleted, freeing up the storage resources occupied by the expired shards in the target sliding window. Finally, after the aggregation processing of the stream data in the target sliding window is completed, step S601 can be executed to receive new stream data, update the target sliding window with the new stream data, and recalculate the aggregation results of the stream data in the moved target sliding window.

[0119] In summary, according to the sliding window-based data processing method of this exemplary embodiment, the target sliding window of the received stream data and the shard position of the stream data within the target sliding window can be determined according to a preset division rule, and the aggregation algorithm is called to aggregate the stream data at each shard position within the target sliding window to obtain an aggregation result. The aggregation result is then used to update the tree structure constructed using the circular array, and the calculation result of the stream data within the target sliding window is determined through the nodes of the tree structure. On the one hand, by calling the aggregation algorithm to aggregate the stream data at each shard position within the target sliding window, the sliding window can be used to implement the aggregation processing of the stream data, and part of the aggregation results of the stream data can be shared between sliding windows, which can reduce the impact of redundant calculations and reduce unnecessary data copy operations. On the other hand, by using the aggregation result to update the tree structure constructed using the circular array, the characteristics of the tree structure can be used to reduce the complexity of stream data processing and improve data processing efficiency.

[0120] The exemplary embodiments of the present disclosure further provide a stream processing system, which may be a window operator in a stream computing framework and may use a sliding window to perform aggregation processing on received stream data. Figure 7A schematic diagram of a stream processing system in this exemplary embodiment is shown. As shown in the figure, the stream processing system 700 may include a determination node 710, a computing node 720 and a storage node 730, wherein the determination node 710 can be used to determine the target sliding window of the received stream data and the shard position of the stream data within the target sliding window according to a preset division rule, the computing node 720 can be used to call an aggregation algorithm to perform aggregation processing on the stream data at each shard position within the target sliding window to obtain an aggregation result, and use the aggregation result to update a tree structure constructed using a circular array, and the storage node 730 can be used to store the tree structure, and the leaf nodes and non-leaf nodes of the tree structure can be used to store the stream data within the target sliding window and the local aggregation results of the stream data within the target sliding window, respectively.

[0121] After receiving the stream data, the determination node 710 can extract key information from the stream data, such as the time and order of receipt of the stream data, and determine the target sliding window of the stream data according to the preset partitioning rules. For example, the target sliding window of the stream data can be determined as the sliding window to which the stream data received at the last moment belongs based on the extracted key information, and the stream data can be divided into corresponding shard positions within the target sliding window according to the sharding rules in the partitioning rules. The storage node 730 can maintain a tree structure that stores the stream data and its aggregation results, and is the core part of implementing window calculations. When performing calculations, the computing node 720 can obtain the stream data in the storage node 730, aggregate the stream data according to the corresponding aggregation algorithm, and then store the calculated aggregation results in the storage node 730. As time changes, the nodes of the tree structure in the storage node 730 will be updated as the aggregation results change.

[0122] In this exemplary embodiment, for each aggregation algorithm, an operator can write an aggregation function to implement it, and when executing the aggregation function, it can be decomposed into primitive operations such as Calculate, Merge, Split, and GetRes.

[0123] Calculate(e:Elements): PartAgg can be used to calculate part of the data within a data shard. Generally, a data shard contains many elements, and the aggregation results of these elements are calculated. The aggregation results can be used to merge with other aggregation results; Merge(x:PartAgg, y:PartAgg) The Merge operation can be expressed as x=x⊕y, which is used to merge two partial aggregation results; Split(x:PartAgg, y:PartAgg) The Split operation can be expressed as x=x⊕y, which separates the result of one partial aggregation from the result of another partial aggregation. It is mostly used in scenarios where some old data is discarded after the window is triggered and the child node data needs to be stripped from the parent node; GetRes(x:PartAgg): The Res GetRes operation can obtain the final aggregation result.

[0124] In a compute node, the following six basic operations can be included. These basic operations can be implemented based on the above four primitive operations:

[0125] The update(index, type) operation updates the associated parent node of the data shard at the specified location.

[0126] add(slice) can find the next insertion position from the loop array and call update to update the corresponding value.

[0127] remove(num) deletes multiple expired shards and calls update to update the corresponding values.

[0128] prefix(index) is used to calculate the prefix sum of the loop array to the end of index.

[0129] suffix(index) is used to calculate the suffix sum of the loop array from index to the end.

[0130] aggregate(head, tail) is called when the window is triggered and calculates the aggregation result from the head pointer to the tail pointer.

[0131] In an optional embodiment, as Figure 8 As shown, the stream processing system 700 may further include a trigger node 740, which may be used to determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time. When the reception time of the stream data is greater than or equal to the dynamic arrival time, the target sliding window of the stream data is triggered. The dynamic arrival time is determined based on the time difference between the maximum event time of the stream data entering the window and the preset delay time.

[0132] After the stream data is input, the stream processing system can process the stream data, extract key information such as the reception time, and determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time. If the reception time of the stream data is greater than or equal to the dynamic arrival time, the trigger node 740 triggers the determination of the target sliding window for the stream data. Specifically, the determination node 710 reads the key information of the stream data from the storage node 730 and determines the target sliding window for the stream data and the shard position of the stream data within the target sliding window according to the preset partitioning rules. After determining the target sliding window, the computing node 720 can call the aggregation algorithm to aggregate the stream data within the target sliding window. At this time, the stream data and the aggregation results obtained by the aggregation processing can be stored in the form of an array in the storage node 730. After the aggregation processing is completed, the output result can be output as an output stream.

[0133] In addition, if Figure 8 As shown, the stream processing system 700 may further include a window node 750 and a function node 760, wherein the window node 750 may set the window type, window length, etc. of each window, and the function node 760 may set a specific window function, such as Max, Min, Count, Sum, etc.

[0134] In this exemplary embodiment, a tree-structured window operator is the core class in the stream processing system. To implement window calculations, determination node 710 can allocate received stream data to target sliding windows using a partitioning operator. The tree-structured window operator includes member variables such as the window size, the window step size, the window start time, the window end time, the window function, and the window aggregation algorithm.

[0135] A tree-structured window operator consists of two important members: a window function and a window aggregation algorithm. A window function can be an interface. Figure 9 A schematic diagram of the classes included in the window operator is shown. Taking standard deviation as an example, standard deviation implements the four basic primitives in the window function interface and can be used to update partial aggregation results in the tree structure. The standard deviation class can contain some basic member variables for calculation, recording information such as the number of all data in the data shard, the cumulative sum, and the square sum. The parent node of the tree structure will store the data of the leaf node. When the data shard newly added to the leaf node is removed from the tree structure, the relevant calculation information of the parent node is updated. At the same time, this calculation information can be directly used to calculate the window aggregation result in the window calculation. The tree structure algorithm implements several basic operations in the window algorithm interface, which are used to update the aggregation tree in the scenario of generating new shards and removing old data, as well as for calculating the aggregation result of the window. The tree structure algorithm contains member variables that store data representing the underlying data structure of the tree structure. The tree structure algorithm can be used to maintain the constructed tree structure.

[0136] Furthermore, this exemplary embodiment also provides a data processing device based on a sliding window, referring to Figure 10 As shown, the sliding window-based data processing device 1000 may include: a first determination module 1010, which can be used to determine the target sliding window of the received stream data and the shard position of the stream data in the target sliding window according to a preset division rule; an aggregation module 1020, which can be used to call an aggregation algorithm to aggregate the stream data at each shard position in the target sliding window to obtain an aggregation result; an update module 1030, which can be used to use the aggregation result to update a tree structure constructed using a circular array, and the leaf nodes and non-leaf nodes of the tree structure can be used to store the stream data in the target sliding window and the local aggregation results of the stream data in the target sliding window, respectively; a second determination module 1040, which can be used to determine the calculation results of the stream data in the target sliding window through the nodes of the tree structure.

[0137] In an exemplary embodiment of the present disclosure, the first determination module 1010 can be used to determine whether the reception time of the stream data is greater than or equal to the dynamic arrival time. The dynamic arrival time is determined based on the time difference between the maximum event time of the stream data entering the window and the preset delay time. When the reception time of the stream data is greater than or equal to the dynamic arrival time, the target sliding window for determining the stream data is triggered.

[0138] In an exemplary embodiment of the present disclosure, the first determination module 1010 can be used to determine the target sliding window of the stream data and the shard position of the stream data within the target sliding window based on the data parameters of the stream data and the window parameters of each window, wherein the data parameters of the stream data include the receiving time and receiving order of the stream data, and the window parameters of each window include the window length and window type of each window.

[0139] In an exemplary embodiment of the present disclosure, the first determination module 1010 can also be used to create a new sliding window when the reception time of the stream data is greater than the window length of each sliding window, and determine the new sliding window as the target sliding window of the stream data; when the reception time of the stream data is not greater than the window length of any sliding window, determine any sliding window as the target sliding window of the stream data.

[0140] In an exemplary embodiment of the present disclosure, each shard position in the target sliding window corresponds to a data shard, and the data shard is composed of one or more stream data.

[0141] In an exemplary embodiment of the present disclosure, when calling an aggregation algorithm to perform aggregation processing on the stream data at each shard position in the target sliding window, the aggregation module 1020 can be used to determine the duplicate shards in the target sliding window when the moving step size of the target sliding window is less than the window length, calculate the aggregation results of other data shards in the target sliding window except for the duplicate shards, and use the aggregation results of the duplicate shards and the aggregation results of the other data shards to determine the aggregation results of the stream data in the target sliding window.

[0142] In an exemplary embodiment of the present disclosure, the aggregation module 1020 can also be used to determine whether the stream data at each shard position in the target sliding window has all arrived when aggregating the stream data at each shard position in the target sliding window. When the stream data at any shard position in the target sliding window has not all arrived, new stream data is received and the new stream data is added to any shard position according to a preset division rule. When the stream data at each shard position in the target sliding window has all arrived, the stream data at each shard position is added to the leaf node of the tree structure.

[0143] In an exemplary embodiment of the present disclosure, after aggregating the stream data at each shard position within the target sliding window, the update module 1030 can be used to update the data shards at each shard position within the target sliding window according to the moving step of the target sliding window, and calculate the aggregation results of the data shards at each shard position within the target sliding window after the data shards are updated.

[0144] In an exemplary embodiment of the present disclosure, the update module 1030 can also be used to delete expired shards within the target sliding window and delete the node data of the expired shards from the tree structure. The expired shards are data shards of the shard positions within the target sliding window that have completed aggregation processing. The module receives new stream data and adds the new stream data to the shard positions corresponding to the expired shards within the target sliding window according to preset division rules.

[0145] In an exemplary embodiment of the present disclosure, the shard range corresponding to the stream data in the target sliding window is located by using the head and tail pointers.

[0146] The specific details of each module in the above device have been described in detail in the implementation method part. The details of the undisclosed scheme can be found in the implementation method part, so they will not be repeated here.

[0147] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0148] The exemplary embodiments of the present disclosure further provide a computer-readable storage medium having stored thereon a program product capable of implementing the methods described above in this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code that, when executed on a terminal device, causes the terminal device to execute the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of the present disclosure.

[0149] refer to Figure 11 As shown, a program product 1100 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described. The program product 1100 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0150] Program product 1100 can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0151] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0153] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0154] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above method. Figure 12 12 is a diagram to describe an electronic device 1200 according to this exemplary embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0155] like Figure 12 As shown, electronic device 1200 may be implemented as a general-purpose computing device. Components of electronic device 1200 may include, but are not limited to, the aforementioned at least one processing unit 1210, the aforementioned at least one storage unit 1220, a bus 1230 connecting various system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0156] The storage unit 1220 stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 executes the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can execute Figures 2 to 3 and Figures 5 and 6 The method steps shown, etc.

[0157] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1221 and / or a cache memory unit 1222 , and may further include a read-only memory unit (ROM) 1223 .

[0158] The storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0159] The bus 1230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0160] The electronic device 1200 can also communicate with one or more external devices 1300 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1200, and / or any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0161] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0162] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0163] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the exemplary embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiments of the present disclosure.

[0164] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and embodiments are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A data processing method based on a sliding window, applied to a stream processing system, characterized in that: The method comprises: Determining a target sliding window of the received stream data and a shard position of the stream data within the target sliding window according to a preset partitioning rule; Calling an aggregation algorithm to aggregate the stream data at each shard position in the target sliding window to obtain an aggregation result; wherein, when the moving step length of the target sliding window is less than the window length, determining the duplicate shards in the target sliding window, calculating the aggregation results of other data shards in the target sliding window except the duplicate shards, and using the aggregation results of the duplicate shards and the aggregation results of the other data shards to determine the aggregation result of the stream data in the target sliding window; Using the aggregation result, a tree structure constructed using a circular array is updated, wherein leaf nodes and non-leaf nodes of the tree structure are used to store the stream data in the target sliding window and the local aggregation result of the stream data in the target sliding window respectively; Determine the calculation result of the stream data in the target sliding window through the nodes of the tree structure; The method further comprises: Determining whether a reception time of the stream data is greater than or equal to a dynamic arrival time, where the dynamic arrival time is determined based on a time difference between a maximum event time of the stream data entering the window and a preset delay time; When the reception time of the stream data is greater than or equal to the dynamic arrival time, a target sliding window for the stream data is triggered to be determined.

2. The method according to claim 1, characterized in that The determining, according to a preset partitioning rule, a target sliding window of the received stream data and a slicing position of the stream data within the target sliding window comprises: Determining a target sliding window for the stream data and a shard position of the stream data within the target sliding window according to data parameters of the stream data and window parameters of each window; The data parameters of the stream data include the receiving time and receiving order of the stream data, and the window parameters of each window include the window length and window type of each window.

3. The method according to claim 2, characterized in that The determining, according to the data parameters of the stream data and the window parameters of each window, a target sliding window of the stream data and a shard position of the stream data within the target sliding window comprises: When the reception time of the stream data is greater than the window length of each sliding window, a new sliding window is created, and the new sliding window is determined as the target sliding window of the stream data; When the reception time of the stream data is not greater than the window length of any sliding window, the any sliding window is determined as the target sliding window of the stream data.

4. The method according to claim 1, wherein Each shard position in the target sliding window corresponds to a data shard, and the data shard is composed of one or more stream data.

5. The method according to claim 1, wherein The method further comprises: When performing aggregation processing on the stream data at each shard position in the target sliding window, determining whether all the stream data at each shard position in the target sliding window has arrived; When the stream data at any shard position in the target sliding window has not all arrived, new stream data is received, and the new stream data is added to any shard position according to the preset division rule; When all the stream data of each shard position in the target sliding window has arrived, the stream data of each shard position is added to the leaf node of the tree structure.

6. The method according to claim 4, characterized in that After aggregating the stream data at each shard position within the target sliding window, the method further includes: Update the data slices at each slice position in the target sliding window according to the moving step of the target sliding window; Calculate the aggregation results of the data shards at each shard position in the target sliding window after the data shards are updated.

7. The method according to claim 6, characterized in that The updating of the data slices at each slice position in the target sliding window according to the moving step of the target sliding window includes: Deleting expired shards in the target sliding window and deleting node data of the expired shards from the tree structure, wherein the expired shards are data shards at shard positions within the target sliding window where aggregation processing has been completed; New stream data is received, and the new stream data is added to the fragment position corresponding to the expired fragment in the target sliding window according to the preset partitioning rule.

8. The method according to claim 1, characterized in that The shard range corresponding to the stream data in the target sliding window is located by using the head and tail pointers.

9. A stream processing system, characterized in that: The system comprises: A determination node is used to determine a target sliding window of the received stream data and a shard position of the stream data within the target sliding window according to a preset partitioning rule; A computing node, configured to call an aggregation algorithm to perform aggregation processing on the stream data at each shard position within the target sliding window, obtain an aggregation result, and use the aggregation result to update a tree structure constructed using a circular array; wherein, when performing the aggregation processing, when the moving step length of the target sliding window is less than the window length, duplicate shards within the target sliding window are determined, aggregation results of other data shards within the target sliding window except the duplicate shards are calculated, and an aggregation result of the stream data within the target sliding window is determined using the aggregation results of the duplicate shards and the aggregation results of the other data shards; A storage node is used to store the tree structure; wherein the leaf nodes and non-leaf nodes of the tree structure are used to store the stream data in the target sliding window and the local aggregation results of the stream data in the target sliding window respectively; The trigger node is configured to execute: Determining whether a reception time of the stream data is greater than or equal to a dynamic arrival time, where the dynamic arrival time is determined based on a time difference between a maximum event time of the stream data entering the window and a preset delay time; When the reception time of the stream data is greater than or equal to the dynamic arrival time, a target sliding window for the stream data is triggered to be determined.

10. A data processing device based on a sliding window, characterized in that: The device comprises: A first determining module, configured to determine a target sliding window of received stream data and a shard position of the stream data within the target sliding window according to a preset partitioning rule; an aggregation module, configured to call an aggregation algorithm to aggregate the stream data at each shard position within the target sliding window to obtain an aggregation result; wherein, when the moving step length of the target sliding window is less than the window length, duplicate shards within the target sliding window are determined, aggregation results of other data shards within the target sliding window except the duplicate shards are calculated, and the aggregation results of the duplicate shards and the aggregation results of the other data shards are used to determine the aggregation result of the stream data within the target sliding window; An updating module, configured to update a tree structure constructed using a circular array using the aggregation result, wherein leaf nodes and non-leaf nodes of the tree structure are respectively used to store the stream data in the target sliding window and the local aggregation result of the stream data in the target sliding window; A second determining module, configured to determine a calculation result of the stream data within the target sliding window through the nodes of the tree structure; The first determining module is configured to execute: Determining whether a reception time of the stream data is greater than or equal to a dynamic arrival time, where the dynamic arrival time is determined based on a time difference between a maximum event time of the stream data entering the window and a preset delay time; When the reception time of the stream data is greater than or equal to the dynamic arrival time, a target sliding window for the stream data is triggered to be determined.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 8 by executing the executable instructions.

Images