Flink large window real-time statistics method and device and storage medium
By nesting custom accumulators in the Flink large window for local aggregation and cleaning the window according to preset rules, the problem of excessive memory resource usage caused by the increase in the time span of large windows in real-time calculations is solved, and more efficient memory management and program stability are achieved.
Patent Information
- Application Number
- CN202411494932.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-03-21
AI Technical Summary
In the Flink real-time computing scenario, as the time span of the time window increases, the amount of state data required to be saved increases significantly, resulting in excessive memory resource usage and even causing memory overflow errors.
By nesting custom accumulators with window functions in the Flink large window, local aggregation of custom windows can be realized, and custom windows in large windows can be sorted and cleaned according to preset rules to reduce memory resource usage.
It effectively reduces the memory resource usage of real-time statistics, avoids memory overflow errors, and further frees up memory space by regularly cleaning out expired or meaningless custom windows, reducing the pressure on the program.
Smart Images
Figure CN119668906A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of real-time data processing technology, and in particular to a Flink large window real-time statistics method, device and storage medium. Background Art
[0002] In Flink real-time computing scenarios, time windows are a key data structure whose core function is to divide data items in a data stream according to the time dimension, thereby achieving real-time statistics of data within a specific time period. This mechanism performs well when processing data with a short time span and can quickly provide accurate statistical results. However, when the time span of the window increases, for example, when statistics are required for the past 10 days, a month, or even half a year, the amount of state data that needs to be saved increases significantly, which can easily lead to excessive memory resource usage and even cause memory overflow errors, ultimately causing the program to crash. Summary of the invention
[0003] In order to solve the above technical problems, the present application provides a Flink large window real-time statistics method, device and storage medium for reducing the memory resource usage of real-time statistics.
[0004] The technical solution provided in this application is described below:
[0005] In a first aspect, the present application provides a Flink large window real-time statistics method, including:
[0006] When it is detected that the target data enters the large window, it is determined whether there is a corresponding target custom window according to the target data. The large window is an outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data.
[0007] If it exists, executing the custom aggregation function to update the target data to the target custom window;
[0008] If it does not exist, creating the target custom window according to the timestamp of the target data, and inputting the target data into the target custom window;
[0009] When there are multiple target custom windows nested in the large window, the target custom windows in the large window are sorted and cleared according to a preset rule.
[0010] Optionally, before judging whether a corresponding target custom window exists according to the target data, the method further includes:
[0011] The timestamps of the target data are aligned according to a preset standard, where the preset standard is a time unit for performing the alignment action.
[0012] Optionally, after aligning the timestamps of the target data according to a preset standard, the method further includes:
[0013] Obtaining the maximum timestamp of the large window;
[0014] Comparing the timestamp of the target data with the maximum timestamp of the large window to obtain a comparison result;
[0015] If the comparison result indicates that the timestamp of the target data is less than or equal to the maximum timestamp of the large window, the maximum timestamp of the large window is kept unchanged;
[0016] If the comparison result indicates that the timestamp of the target data is greater than the maximum timestamp of the large window, the maximum timestamp of the large window is updated to the timestamp of the target data.
[0017] Optionally, after the step of keeping the maximum timestamp of the large window unchanged, the method further includes:
[0018] Calculate the difference between the maximum timestamp of the large window and the timestamp of the target data to obtain a difference calculation result;
[0019] Comparing the difference calculation result with a preset threshold, the preset threshold being the maximum time allowed for target data to be late, and the unit of the preset threshold conforming to the preset standard;
[0020] If the difference calculation result is less than or equal to the preset threshold, executing the step of determining whether there is a corresponding target custom window according to the target data;
[0021] If the difference calculation result is greater than the preset threshold, the target data is discarded.
[0022] Optionally, the preset rule includes a preset time, and the preset time is the length of time that the target data is allowed to stay in the large window. Arranging and clearing the target custom window in the large window according to the preset rule includes:
[0023] Arrange the target custom window in the large window according to preset rules;
[0024] Traversing the timestamps of all the target custom windows within the large window;
[0025] Calculate the difference between the maximum timestamp of the large window and the timestamp of each target custom window respectively to obtain multiple difference values;
[0026] Obtaining a target custom window corresponding to a difference value greater than the preset time among the multiple differences;
[0027] The corresponding target custom window is removed.
[0028] Optionally, the target data further includes a unique ID, and executing the custom aggregate function to update the target data to the target custom window includes:
[0029] According to the target custom window, a pre-aggregation result is obtained in an HBase table, where the HBase table is used to store a result obtained by the target custom window executing the custom aggregation function and a unique ID of the target data, and the pre-aggregation result is a result obtained by the target custom window executing the custom aggregation function before the target data enters the large window;
[0030] Determine whether the unique ID of the target data exists in the HBase table;
[0031] If yes, discard the target data;
[0032] If not, executing the custom aggregation function on the target data and the previous aggregation result to obtain a new aggregation result;
[0033] The new aggregation result replaces the previous aggregation result, and the new aggregation result is stored in the HBase table to update the target custom window.
[0034] Optionally, arranging and clearing the target custom window in the large window according to a preset rule includes:
[0035] Get the target custom window with the latest timestamp;
[0036] Adding all target custom windows except the target custom window with the latest timestamp to the target custom window with the latest timestamp;
[0037] The target custom window within the large window is cleared according to a preset rule.
[0038] A second aspect of the present application provides a Flink large window real-time statistics device, including:
[0039] A custom window unit is used to determine whether there is a corresponding target custom window according to the target data when it is detected that the target data enters the large window. The large window is an outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes data content and a timestamp corresponding to the data.
[0040] A window updating unit, configured to execute the custom aggregation function to update the target data to the target custom window if the target custom window exists;
[0041] a creating unit, configured to create the target custom window according to the timestamp of the target data and input the target data into the target custom window if the custom window does not exist;
[0042] The arranging and cleaning unit is used to arrange and clean the target custom windows in the large window according to preset rules when there are multiple target custom windows nested in the large window.
[0043] A third aspect of the present application provides a device for real-time statistics of a large Flink window, the device comprising:
[0044] Processor, memory, input-output unit, and bus;
[0045] The processor is connected to the memory, the input and output unit, and the bus;
[0046] The memory stores a program, and the processor calls the program to execute the first aspect and the Flink large window real-time statistics method optionally included in any one of the first aspects.
[0047] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method for real-time statistics of a large Flink window according to the first aspect and any optional item in the first aspect is executed.
[0048] It can be seen from the above technical solutions that this application has the following advantages:
[0049] The present application provides a method for real-time statistics of Flink large windows. First, when the target data is detected to enter the large window, it is determined whether there is a corresponding target custom window according to the target data. The large window is the outer window in the Flink framework. The target custom window is an accumulator with window function and custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data; if it exists, the custom aggregation function is executed to update the target data to the target custom window; if it does not exist, the target custom window is created according to the timestamp of the target data, and the target data is input into the target custom window; finally, when the number of target custom windows nested in the large window is multiple, the target custom windows in the large window are sorted and cleaned according to the preset rules. The present application realizes the local aggregation of the custom window by nesting an accumulator with window function in the Flink large window, and then retains the aggregation result according to the specified timestamp. Compared with the prior art of retaining each original data, the memory resources occupied by the storage data of the present application are greatly reduced. And regular cleaning of expired or meaningless custom windows can further free up memory space and reduce the pressure of the program. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0051] Figure 1 A flowchart of an embodiment of the Flink large window real-time statistics method provided in this application;
[0052] Figure 2 A flowchart of another embodiment of the Flink large window real-time statistics method provided in this application;
[0053] Figure 3 A flowchart of another embodiment of the Flink large window real-time statistics method provided in this application;
[0054] Figure 4 A flowchart of another embodiment of the Flink large window real-time statistics method provided in this application;
[0055] Figure 5 A flowchart of another embodiment of the Flink large window real-time statistics method provided in this application;
[0056] Figure 6 A flowchart of another embodiment of the Flink large window real-time statistics method provided in this application;
[0057] Figure 7 A schematic diagram of the structure of an embodiment of a device for real-time statistics of a large Flink window provided in this application;
[0058] Figure 8 This is a schematic diagram of the structure of another embodiment of the Flink large window real-time statistics device provided in this application. DETAILED DESCRIPTION
[0059] The present application provides a Flink large window real-time statistics method, device and storage medium for reducing the memory resource usage of real-time statistics.
[0060] It should be noted that the Flink large window real-time statistics method provided in this application can be applied to terminals, systems, and servers. For example, the terminal can be a smart phone or computer, tablet computer, smart TV, smart watch, portable computer terminal, or a fixed terminal such as a desktop computer.
[0061] It should also be noted that the terms "first", "second", etc. in the specification and drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order, sequence or importance.
[0062] See also Figure 1 The present application provides an embodiment of a Flink large window real-time statistics method, and the Flink large window real-time statistics method includes:
[0063] Step S101: when it is detected that the target data enters the large window, it is determined whether there is a corresponding target custom window according to the target data. The large window is the outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data. If it exists, step S102 is executed; if not, step S103 is executed.
[0064] Step S102, executing the custom aggregation function to update the target data to the target custom window;
[0065] Step S103, creating a target custom window according to the timestamp of the target data, and inputting the target data into the target custom window;
[0066] Step S104: when there are multiple target custom windows nested in the large window, sort and clear the target custom windows in the large window according to a preset rule.
[0067] The method of real-time statistics of Flink large windows of the present invention, first, when it is detected that the target data enters the large window, it is determined whether there is a corresponding target custom window according to the target data, the large window is the outer window in the Flink framework, the target custom window is an accumulator with window function and custom aggregation function nested in the large window, and the target data includes the data content and the timestamp corresponding to the data; if it exists, the custom aggregation function is executed to update the target data to the target custom window; if it does not exist, the target custom window is created according to the timestamp of the target data, and the target data is input into the target custom window; finally, when the number of target custom windows nested in the large window is multiple, the target custom windows in the large window are sorted and cleaned according to the preset rules. The present application realizes the local aggregation of the custom window by nesting an accumulator with a window function in the Flink large window, and then retains the aggregation result according to the specified timestamp. Compared with the prior art of retaining each original data, the memory resources occupied by the storage data of the present application are greatly reduced. And regular cleaning of expired or meaningless custom windows can further release memory space and reduce the pressure of the program.
[0068] In step S101 of some embodiments, when it is detected that the target data enters the large window, it is determined whether there is a corresponding target custom window according to the target data. The large window is an outer window in the Flink framework, and the target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data. Adding new data to the large window is one of the core methods of Flink real-time computing. When there are at least two or more newly added data (i.e., target data) entering, these target data will be counted in real time. Among them, the target data includes a timestamp (i.e., the time when the event corresponding to the target data occurs) and specific data content. In the prior art, Flink real-time statistics usually aggregate all data in the same time window according to the aggregation function. When the amount of data is too large, the program or system that executes the aggregation function is prone to crash due to excessive computing pressure. In order to solve the above problem, in this embodiment, multiple custom windows are nested in the large window, so that the data is first locally aggregated through the custom window. Since the custom window of this embodiment is an accumulator with window statistics and storage functions, the large window can obtain the local aggregation results of multiple custom windows, and then perform outer aggregation, that is, use the custom window as the accumulator of the outer aggregation function to generate the final statistical result, thereby reducing the computational pressure of the large window.
[0069] Reference Figure 2 Before processing the target data, the timestamps of the target data and the large window should be sorted to facilitate subsequent operations based on the timestamps, which may include, but are not limited to, the following:
[0070] Step S201, aligning the timestamps of the target data according to a preset standard, where the preset standard is the time unit for performing the alignment action;
[0071] Step S202, obtaining the maximum timestamp of the large window;
[0072] Step S203, comparing the timestamp of the target data with the maximum timestamp of the large window to obtain a comparison result;
[0073] Step S204, if the comparison result indicates that the timestamp of the target data is less than or equal to the maximum timestamp of the large window, the maximum timestamp of the large window is kept unchanged;
[0074] Step S205: If the comparison result indicates that the timestamp of the target data is greater than the maximum timestamp of the large window, the maximum timestamp of the large window is updated to the timestamp of the target data.
[0075] In steps S201 to S205 of some embodiments, the timestamp of the target data is first aligned according to a preset standard, and the preset standard is the time unit for performing the alignment action; then the maximum timestamp of the large window is obtained; then the timestamp of the target data is compared with the maximum timestamp of the large window to obtain a comparison result; if the comparison result indicates that the timestamp of the target data is less than or equal to the maximum timestamp of the large window, the maximum timestamp of the large window is kept unchanged; if the comparison result indicates that the timestamp of the target data is greater than the maximum timestamp of the large window, the maximum timestamp of the large window is updated to the timestamp of the target data. For each target data entering the large window, it should be adjusted according to its timestamp and the preset standard to ensure that all timestamps conform to a unified time unit format. The preset standard is usually a time unit, such as seconds, minutes, hours, days, etc. The timestamps of all targets should be aligned to the same time axis to ensure the consistency and comparability of the data. And the timestamp alignment should be as accurate as possible to reduce the error caused by time asynchrony. At the same time, no valid data should be lost during the alignment process, unless some data needs to be eliminated in specific circumstances (such as poor data quality). The alignment operation is to adjust the timestamp to the closest preset time unit point. For example, if the preset standard is minutes, then a time point "14:03:27" may be aligned to "14:03:00". More specifically, if the preset standard in this embodiment is days, when the timestamp of the target data is 1724155541251 (2024-08-2020:05:41), the timestamp will be aligned to 1724083200000 (2024-08-20 00:00:00). It should be noted that this embodiment sets the preset standard by setting the compareTimeUnit field.
[0076] After aligning the timestamp of the target data, the maximum timestamp of the large window should also be set according to the timestamp of the target data. The maximum timestamp can only increase but not decrease. The maximum timestamp of the large window is used to determine whether the target data is expired or late. If the timestamp of the latest target data entering the large window is greater than the maximum timestamp of the current large window, the timestamp of the target data is updated to the maximum timestamp of the large window. If the timestamp of the target data is not greater than the maximum timestamp of the large window, that is, the target data belongs to the time range of the current large window or is late / expired data, there is no need to update the maximum timestamp of the large window, and the maximum timestamp of the large window can be kept unchanged.
[0077] Reference Figure 3 In practice, due to the large amount of data that needs to be counted in real time, the system may be busy, causing data to enter the large window late. Figure 2 In the illustrated embodiment, the processing of data delayed in entering the large window may specifically include, but is not limited to, the following:
[0078] Step S301, calculating the difference between the maximum timestamp of the large window and the timestamp of the target data to obtain a difference calculation result;
[0079] Step S302, comparing the difference calculation result with a preset threshold, where the preset threshold is the maximum time allowed for the target data to be late, and the unit of the preset threshold meets the preset standard;
[0080] Step S303, if the difference calculation result is less than or equal to the preset threshold, then executing the step of determining whether there is a corresponding target custom window according to the target data;
[0081] Step S304: If the difference calculation result is greater than a preset threshold, the target data is discarded.
[0082] In some embodiments, in step S301 to step S304, the difference between the maximum timestamp of the large window and the timestamp of the target data is first calculated to obtain a difference calculation result; then the difference calculation result is compared with a preset threshold, where the preset threshold is the maximum time that the target data is allowed to be late, and the unit of the preset threshold meets the preset standard; if the difference calculation result is less than or equal to the preset threshold, the step of determining whether there is a corresponding target custom window based on the target data is executed; if the difference calculation result is greater than the preset threshold, the target data is discarded. In this embodiment, according to Figure 2In the embodiment shown, if the timestamp of the target data is not greater than the maximum timestamp of the large window, the target data may belong to the time range of the current large window, or may be late data, so the above-mentioned multiple situations should be judged. A maximum time allowed for the target data to be late is preset, that is, a preset threshold. In this embodiment, the preset threshold is the maximum disorder duration of the large window, and the preset threshold is used as the basis for judgment. Then, the difference between the maximum time of the large window and the timestamp of the target data is calculated, that is, the difference calculation result, and then the difference calculation result is compared with the preset threshold.
[0083] If the difference calculation result is less than or equal to the preset threshold, it means that although the target data is late, it is still valid data within the time range allowed for lateness, and the target custom window corresponding to the target data should be further searched to allow the target data to be counted. If the difference calculation result is greater than the preset threshold, it means that the target data is late and is not within the allowed lateness range, the target data is invalid, and the target data can be directly discarded. This ensures that the data participating in the internal aggregation of the custom window are all valid data.
[0084] Specifically, for example: 1. If the maximum disorder duration of the large window is preset to 2 days, then the preset threshold is 2 days, and the current large window time is August 7, 2024, and the target data time is August 6, 2024, then the target data is 1 day late, and the late time is less than the preset threshold, so the target data is valid, and the target custom window corresponding to the target data is further queried. 2. If the preset threshold is 2 days, and the current large window time is August 8, 2024, and the target data time is August 5, 2024, then the target data is 3 days late, and the late time is greater than the preset threshold, so the target data is invalid and is directly discarded.
[0085] In step S102 of some embodiments, after determining whether there is a corresponding target custom window according to the target data in step S101, if the corresponding target custom window exists, a custom aggregation function is executed to update the target data to the target custom window. Step S102 is intended to achieve local aggregation of the target custom window. If a corresponding target custom window is found according to the target data, it means that the target data meets the conditions of the corresponding target custom window, and the target data can be input into the corresponding target custom window, so that the target custom window can include the target data in the statistics by executing the custom aggregation function, and update the internal data of the target custom window.
[0086] Reference Figure 4 According to some embodiments of the present invention, the target data also includes a unique ID, and executing a custom aggregate function to update the target data to the target custom window may specifically include, but is not limited to, the following:
[0087] Step S401, according to the target custom window, obtain the pre-aggregation result in the HBase table, the HBase table is used to store the result obtained by executing the custom aggregation function of the target custom window and the unique ID of the target data, the pre-aggregation result is the result obtained by executing the custom aggregation function of the target custom window before the target data enters the large window;
[0088] Step S402, determine whether there is a unique ID of the target data in the HBase table; if so, execute step S403; if not, execute step S404;
[0089] Step S403, discarding target data;
[0090] Step S404, executing a custom aggregation function on the target data and the previous aggregation result to obtain a new aggregation result;
[0091] Step S405: Replace the previous aggregation result with the new aggregation result, and store the new aggregation result in the HBase table to update the target custom window.
[0092] In steps S401 to S405 of some embodiments, first, according to the target custom window, the pre-aggregation result is obtained in the HBase table. The HBase table is used to store the result obtained by the target custom window executing the custom aggregation function and the unique ID of the target data. The pre-aggregation result is the result obtained by the target custom window executing the custom aggregation function before the target data enters the large window; then, it is determined whether the unique ID of the target data exists in the HBase table; if so, the target data is discarded; if not, the custom aggregation function is executed on the target data and the pre-aggregation result to obtain a new aggregation result; then the new aggregation result replaces the pre-aggregation result, and the new aggregation result is stored in the HBase table to update the target custom window. The HBase table is a basic component unit in the HBase system, which is used to store and manage data. It can store and process time series data, and provide low-latency data access, which is suitable for real-time data processing. This embodiment uses the HBase table to store the result obtained by the target custom window executing the custom aggregation function and the unique ID of the target data, which can reduce the risk of data loss and greatly reduce the memory resource usage of the large window. It should be noted that after a series of preliminary data screening and processing, each time new target data is added to the target custom window, the execution of the custom aggregate function is automatically triggered unconditionally.
[0093] Before executing a custom aggregation function, you should also determine whether the target data is repeated statistical data, that is, determine it through the unique ID carried by the target data, and query whether there is the same unique ID in the HBase table. If yes, it means that the target data is repeated data and should be discarded directly; if not, it means that the target data is new data and should participate in local aggregation. The target data participates in local aggregation, that is, first find the previous aggregation result in the HBase table, then execute the custom aggregation function to count the previous aggregation result and the target data, obtain the new aggregation result, and finally update and replace the previous aggregation result with the new aggregation result, and store it in the HBase table for subsequent use or query. For example, if the target data time is 20:30:00 on August 1, 2024, and the data content of the target data is "payment amount: 34 yuan", and the previous aggregation result of the corresponding target custom window is the statistical result "payment amount: 105 yuan" from "00:00:00 on August 1, 2024" to "16:30:00 on August 1, 2024", then after executing the custom aggregation function on the target data and the previous aggregation result, the new aggregation result is the statistical result "payment amount: 139 yuan" from "00:00:00 on August 1, 2024" to "20:30:00 on August 1, 2024". Then the new aggregation result is updated to replace the previous aggregation result, stored in the HBase table, and becomes the previous aggregation result participating in the next execution of the custom aggregation function. The same applies to other cases.
[0094] In step S103 of some embodiments, after step S101 determines whether there is a corresponding target custom window according to the target data, if the corresponding target custom window does not exist, a target custom window is created according to the timestamp of the target data, and the target data is entered into the target custom window. The non-existence of the corresponding target custom window means that there is no custom window that meets the timestamp of the target data, and a new target custom window needs to be created to meet the corresponding timestamp. For example, the target custom windows nested in the current large window are only: Custom Window 1 (time is August 1, 2024) and Custom Window 2 (time is August 2, 2024), and the target custom window only allows the input of target data of the same date. The target data entering the large window on August 3, 2024 does not have a corresponding target custom window, and Custom Window 3 (time is August 3, 2024) should be created.
[0095] In step S104 of some embodiments, when there are multiple target custom windows nested in the large window, the target custom windows in the large window are sorted and cleaned according to preset rules. There are multiple target custom windows nested in the large window, which means that multiple target custom windows have been generated according to the timestamp, and each target custom window has stored its own local aggregate data. As the outer aggregation function nested in the large window, the multiple target custom windows should also be adjusted by the outer aggregation function of the large window to realize the real-time statistical data of the Flink large window. At the same time, in order to reduce the problem of excessive resource occupation caused by expired data, the target custom windows in the large window should be cleaned up regularly, that is, the expired target custom windows should be discarded according to the aligned timestamps and preset time, thereby releasing memory resources.
[0096] Reference Figure 5 , arranging and cleaning the target custom window in the large window according to the preset rules may specifically include, but not limited to, the following:
[0097] Step S501, obtaining the target custom window with the latest timestamp;
[0098] Step S502, adding all target custom windows except the target custom window with the latest timestamp to the target custom window with the latest timestamp;
[0099] Step S503, clearing the target custom window in the large window according to a preset rule.
[0100] In steps S501 to S503 of some embodiments, the target custom window with the latest timestamp is first obtained; then all target custom windows except the target custom window with the latest timestamp are added to the target custom window with the latest timestamp, and finally the target custom windows in the large window are cleaned up according to preset rules. All valid target custom windows nested in the large window are traversed, and the target custom window corresponding to the latest timestamp is filtered out according to the aligned timestamps, and then the remaining target custom windows are added to the target custom window corresponding to the latest timestamp, that is, in the HBase table, the data of the target custom window corresponding to the latest timestamp includes its own aggregation results and the aggregation results of the remaining target custom windows. Step S503 in this embodiment is the same as the aforementioned Figure 1 Step S104 in the illustrated embodiment is similar and will not be described in detail here.
[0101] For example: In case ①, the maximum time of the current large window is August 2, 2024. All valid target custom windows nested in the large window are: Custom Window 1 and Custom Window 2. Among them, the aggregation result of Custom Window 1 itself is "Payment amount on August 1, 2024: 50 yuan", and the aggregation result of Custom Window 2 itself is "Payment amount on August 2, 2024: 65 yuan". Then Custom Window 2 is the latest target custom window. Custom Window 1 is added to Custom Window 2. In the HBase table, the data of Custom Window 2 includes: "Payment amount on August 1, 2024: 50 yuan" and "Payment amount on August 2, 2024: 65 yuan". Case 2: Based on case 1, when the maximum time of the large window is updated to August 3, 2024, and a new nested custom window 3 is added, the aggregation result of custom window 3 is "payment amount on August 3, 2024: 107 yuan", then custom window 3 is the latest target custom window at this time, and custom window 1 and custom window 2 are added to custom window 3 respectively. In the HBase table, the data of custom window 3 includes: "payment amount on August 1, 2024: 50 yuan", "payment amount on August 2, 2024: 65 yuan" and "payment amount on August 3, 2024: 107 yuan". Other cases are similar.
[0102] Reference Figure 6 According to some embodiments of the present invention, the preset rule includes a preset time, which is the length of time that the target data is allowed to stay in the large window. The target custom window in the large window is periodically cleared according to the preset time, and specifically includes, but is not limited to, the following:
[0103] Step S601, arranging the target custom windows in the large window according to preset rules;
[0104] Step S602, traversing the timestamps of all target custom windows in the large window;
[0105] Step S603, respectively calculating the difference between the maximum time of the large window and the timestamp of each target custom window to obtain multiple difference values;
[0106] Step S604, obtaining a target custom window corresponding to a difference value greater than a preset time among a plurality of differences;
[0107] Step S605: remove the corresponding target custom window.
[0108] In some embodiments, in steps S601 to S605, the target custom windows in the large window are first sorted according to preset rules; then the timestamps of all target custom windows in the large window are traversed; then the difference between the maximum time of the large window and the timestamp of each target custom window is calculated to obtain multiple differences; then the target custom window corresponding to the difference greater than the preset time among the multiple differences is obtained; finally, the corresponding target custom window is removed. Step S601 in this embodiment is the same as the above Figure 1 Step S104 in the illustrated embodiment is similar and will not be described here in detail. Perform a traversal operation, i.e., align and obtain the timestamps of all target custom windows. Then calculate the time difference between each target custom window and the maximum time of the large window, and determine whether the target custom window is expired based on the time difference and the preset time. If the determination result indicates that the target custom window is expired, remove the expired target custom window from the large window, and delete all the data of the expired target custom window stored in the HBase table. Thus, the resource usage is further reduced and the computing pressure of the program is reduced.
[0109] For example, the preset time is set to 5 days, that is, the length of time that the large window allows the target data to stay in the large window, and the maximum time of the current large window is August 6, 2024. After traversal, the target custom windows nested in the current large window are: custom window 1 (corresponding timestamp is August 1, 2024), custom window 2 (corresponding timestamp is August 2, 2024), custom window 3 (corresponding timestamp is August 3, 2024), custom window 4 (corresponding timestamp is August 4, 2024), custom window 5 (corresponding timestamp is August 5, 2024) and custom window 6 (corresponding timestamp is August 6, 2024). Then, the difference between the maximum time of the large window and the time of each target custom window is calculated. It can be seen that custom window 1 has expired, so custom window 1 is removed from the large window, and all data of custom window 1 in the HBase table is deleted, including data added to custom windows 2 to 6.
[0110] The above embodiment describes the Flink large window real-time statistics method provided by the present application. The following describes the Flink large window real-time statistics device and storage medium provided by the present application:
[0111] See also Figure 7 The Flink large window real-time statistics device includes:
[0112] The custom window unit 701 is used to determine whether there is a corresponding target custom window according to the target data when it is detected that the target data enters the large window. The large window is an outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data.
[0113] The window updating unit 702 is used to execute the custom aggregation function to update the target data to the target custom window if the target custom window exists;
[0114] A creation unit 703, for creating a target custom window according to the timestamp of the target data if the custom window does not exist, and inputting the target data into the target custom window;
[0115] The arranging and clearing unit 704 is used to arrange and clear the target custom windows in the large window according to a preset rule when there are multiple target custom windows nested in the large window.
[0116] Optionally, before the custom window unit 701, it also includes:
[0117] The alignment unit 705 is used to align the timestamps of the target data according to a preset standard, where the preset standard is a time unit for performing the alignment action.
[0118] Optionally, after the alignment unit 705, the following is further included:
[0119] A maximum timestamp unit 706, used to obtain a maximum timestamp of a large window;
[0120] A comparison unit 707, used to compare the timestamp of the target data with the maximum timestamp of the large window to obtain a comparison result;
[0121] A holding unit 708, configured to keep the maximum timestamp of the large window unchanged if the comparison result indicates that the timestamp of the target data is less than or equal to the maximum timestamp of the large window;
[0122] The time updating unit 709 is configured to update the maximum timestamp of the large window to the timestamp of the target data if the comparison result indicates that the timestamp of the target data is greater than the maximum timestamp of the large window.
[0123] Optionally, after the maintaining unit 708, the method further includes:
[0124] A calculation unit 710 is used to calculate the difference between the maximum timestamp of the large window and the timestamp of the target data to obtain a difference calculation result;
[0125] A threshold comparison unit 711 is used to compare the difference calculation result with a preset threshold, where the preset threshold is the maximum time allowed for target data to be late, and the unit of the preset threshold meets the preset standard;
[0126] An execution unit 712 is configured to execute a step of determining whether there is a corresponding target custom window according to the target data if the difference calculation result is less than or equal to a preset threshold;
[0127] The discarding unit 713 is configured to discard the target data if the difference calculation result is greater than a preset threshold.
[0128] Optionally, the preset rule includes a preset time, and the preset time is the length of time that the target data is allowed to stay in the large window. The sorting and cleaning unit 704 is specifically used for:
[0129] Arrange the target custom windows within the large window according to preset rules;
[0130] Traverse the timestamps of all target custom windows within the large window;
[0131] Calculate the difference between the maximum timestamp of the large window and the timestamp of each target custom window to obtain multiple difference values;
[0132] Obtain a target custom window corresponding to a difference value greater than a preset time among multiple differences;
[0133] Remove the corresponding target customization window.
[0134] Optionally, the target data further includes a unique ID, and the window updating unit 702 is specifically configured to:
[0135] According to the target custom window, obtain the pre-aggregation result in the HBase table. The HBase table is used to store the result obtained by executing the custom aggregation function in the target custom window and the unique ID of the target data. The pre-aggregation result is the result obtained by executing the custom aggregation function in the target custom window before the target data enters the large window.
[0136] Determine whether the unique ID of the target data exists in the HBase table;
[0137] If so, the target data is discarded;
[0138] If not, the custom aggregation function is executed on the target data and the previous aggregation result to obtain a new aggregation result;
[0139] Replace the previous aggregation result with the new aggregation result and store the new aggregation result in the HBase table to update the target custom window.
[0140] Optionally, the sorting and cleaning unit 704 is specifically used for:
[0141] Get the target custom window with the latest timestamp;
[0142] Add all target custom windows except the target custom window with the latest timestamp to the target custom window with the latest timestamp;
[0143] Clean the target custom window within the large window according to the preset rules.
[0144] This application also provides a Flink large window real-time statistics device, please refer to Figure 8 , the device comprises:
[0145] Processor 801, memory 802, input and output unit 803, bus 804;
[0146] The processor 801 is connected to the memory 802, the input and output unit 803 and the bus 804;
[0147] The memory 802 stores a program, and the processor 801 calls the program to execute the following steps: Figures 1 to 6 The Flink large window real-time statistics method shown in any one of the embodiments.
[0148] The present application also relates to a computer-readable storage medium, on which a program is stored, and when the program is run on a computer, the computer executes the following Figures 1 to 6 The Flink large window real-time statistics method shown in any one of the embodiments.
[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0150] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
Claims
1. A Flink large window real-time statistics method, characterized in that: The method comprises: When it is detected that the target data enters the large window, it is determined whether there is a corresponding target custom window according to the target data. The large window is an outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes the data content and the timestamp corresponding to the data. If it exists, executing the custom aggregation function to update the target data to the target custom window; If it does not exist, creating the target custom window according to the timestamp of the target data, and inputting the target data into the target custom window; When there are multiple target custom windows nested in the large window, the target custom windows in the large window are sorted and cleared according to a preset rule.
2. The method according to claim 1, characterized in that Before judging whether there is a corresponding target custom window according to the target data, the method further includes: The timestamps of the target data are aligned according to a preset standard, where the preset standard is a time unit for performing the alignment action.
3. The method according to claim 2, characterized in that After aligning the timestamps of the target data according to the preset standard, the method further includes: Obtaining the maximum timestamp of the large window; Comparing the timestamp of the target data with the maximum timestamp of the large window to obtain a comparison result; If the comparison result indicates that the timestamp of the target data is less than or equal to the maximum timestamp of the large window, the maximum timestamp of the large window is kept unchanged; If the comparison result indicates that the timestamp of the target data is greater than the maximum timestamp of the large window, the maximum timestamp of the large window is updated to the timestamp of the target data.
4. The method according to claim 3, characterized in that After the step of keeping the maximum timestamp of the large window unchanged, the method further includes: Calculate the difference between the maximum timestamp of the large window and the timestamp of the target data to obtain a difference calculation result; Comparing the difference calculation result with a preset threshold, the preset threshold being the maximum time allowed for target data to be late, and the unit of the preset threshold conforming to the preset standard; If the difference calculation result is less than or equal to the preset threshold, executing the step of determining whether there is a corresponding target custom window according to the target data; If the difference calculation result is greater than the preset threshold, the target data is discarded.
5. The method according to claim 3, characterized in that The preset rule includes a preset time, and the preset time is the length of time that the target data is allowed to stay in the large window. Arranging and cleaning the target custom window in the large window according to the preset rule includes: Arrange the target custom window in the large window according to preset rules; Traversing the timestamps of all the target custom windows within the large window; Calculate the difference between the maximum timestamp of the large window and the timestamp of each target custom window respectively to obtain multiple difference values; Obtaining a target custom window corresponding to a difference value greater than the preset time among the multiple differences; The corresponding target custom window is removed.
6. The method according to claim 1, characterized in that The target data also includes a unique ID, and executing the custom aggregate function to update the target data to the target custom window includes: According to the target custom window, a pre-aggregation result is obtained in an HBase table, where the HBase table is used to store a result obtained by the target custom window executing the custom aggregation function and a unique ID of the target data, and the pre-aggregation result is a result obtained by the target custom window executing the custom aggregation function before the target data enters the large window; Determine whether the unique ID of the target data exists in the HBase table; If yes, discard the target data; If not, executing the custom aggregation function on the target data and the previous aggregation result to obtain a new aggregation result; The new aggregation result replaces the previous aggregation result, and the new aggregation result is stored in the HBase table to update the target custom window.
7. The method according to any one of claims 1 to 6, characterized in that Arranging and clearing the target custom window in the large window according to preset rules includes: Get the target custom window with the latest timestamp; Adding all target custom windows except the target custom window with the latest timestamp to the target custom window with the latest timestamp; The target custom window within the large window is cleared according to a preset rule.
8. A device for real-time statistics of a large Flink window, characterized in that: The device comprises: A custom window unit is used to determine whether there is a corresponding target custom window according to the target data when it is detected that the target data enters the large window. The large window is an outer window in the Flink framework. The target custom window is an accumulator with a window function and a custom aggregation function nested in the large window. The target data includes data content and a timestamp corresponding to the data. A window updating unit, configured to execute the custom aggregation function to update the target data to the target custom window if the target custom window exists; a creating unit, configured to create the target custom window according to the timestamp of the target data and input the target data into the target custom window if the custom window does not exist; The arranging and cleaning unit is used to arrange and clean the target custom windows in the large window according to preset rules when there are multiple target custom windows nested in the large window.
9. A device for real-time statistics of a large Flink window, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the Flink large window real-time statistics method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a program stored thereon, wherein when the program is executed on a computer, the Flink large window real-time statistics method according to any one of claims 1 to 7 is executed.