A dynamic window adjustment method and device

Abstract

The present invention provides a dynamic window adjustment method and device, the method comprising: when performing real-time calculation in the current trigger time window, calculating the message efficiency up to the current time; judging whether the message efficiency is greater than a preset message efficiency threshold; if so, obtaining the tolerable disorder time of the next trigger time window by reducing the tolerable disorder time of the current trigger time window; if not, obtaining the tolerable disorder time of the next trigger time window by increasing the tolerable disorder time of the current trigger time window. The present invention can dynamically adjust the tolerable disorder time according to the message efficiency threshold set by the user when performing real-time calculation in each trigger time window, which can adapt to the tolerable disorder time of different application scenarios, thereby achieving the purpose of dynamically adjusting the time window trigger time, completing the statistical analysis of the time window in a timely manner, and improving the timeliness and accuracy of the real-time calculation.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more specifically, to a dynamic window adjustment method and device. Background Art

[0002] In the field of real-time computing, real-time statistical analysis is of great significance for users to understand real-time information in a timely manner. Due to the real-time requirements of real-time computing, the limitations of system resources, and the infinity of stream messages, it is impossible to analyze and count like offline data. Therefore, the messages used for real-time statistical analysis are obtained based on the time window, and users complete statistical analysis based on the messages within the time window.

[0003] At present, time windows are mainly divided based on message timestamps (Event-Time, the timestamp provided by the message itself). In order to avoid message disorder caused by factors such as network anomalies and terminal anomalies, watermark timestamps are often introduced in real-time calculations to mark the time when the message is due. If the message timestamp of the subsequent received message is less than the watermark timestamp, it will be discarded. Otherwise, the calculation of the time window will be triggered.

[0004] However, the watermark timestamp is usually obtained by subtracting the fixed tolerable out-of-order time from the maximum message timestamp of the messages received after the time window. If the tolerable out-of-order time is set too large, the requirements for system resources will be high and the acquisition of real-time calculation results will be slower. If the tolerable out-of-order time is set too small, a large number of messages will be discarded due to delays, resulting in large deviations in statistical results. Summary of the invention

[0005] In view of this, in order to solve the above problems, the present invention provides a dynamic window adjustment method and device.

[0006] The technical solution is as follows:

[0007] A dynamic window adjustment method, the method comprising:

[0008] When performing real-time calculations in this trigger time window, the efficiency of the message up to the current time is calculated;

[0009] Determining whether the message effectiveness is greater than a preset message effectiveness threshold;

[0010] If the message efficiency is greater than the message efficiency threshold, the tolerable out-of-order time of the next trigger time window is obtained by reducing the tolerable out-of-order time of the current trigger time window;

[0011] If the message efficiency is not greater than the message efficiency threshold, the tolerable out-of-order time of the next triggering time window is obtained by increasing the tolerable out-of-order time of the current triggering time window.

[0012] Preferably, the calculating of the message efficiency up to the present time includes:

[0013] Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window;

[0014] The message efficiency U up to the current time is calculated according to the following formula n :

[0015]

[0016] Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, D ui The number of invalid messages within the calculation time of the i-th trigger time window in history.

[0017] Preferably, the step of obtaining the tolerable out-of-order time of the next trigger time window by reducing the tolerable out-of-order time of the current trigger time window includes:

[0018] Reduce the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0019]

[0020] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and ceil() means rounding up.

[0021] Preferably, the step of obtaining the tolerable out-of-order time of the next trigger time window by increasing the tolerable out-of-order time of the current trigger time window includes:

[0022] Increase the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0023]

[0024] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and floor() means rounding down.

[0025] A dynamic window adjustment device, the device comprising:

[0026] The first calculation module is used to calculate the message efficiency up to the current time when performing real-time calculation in this trigger time window;

[0027] A judging module, used to judge whether the message efficiency is greater than a preset message efficiency threshold;

[0028] A second calculation module is used for obtaining the tolerable out-of-order time of the next triggering time window by reducing the tolerable out-of-order time of the current triggering time window if the message efficiency is greater than the message efficiency threshold;

[0029] The third calculation module is used to obtain the tolerable out-of-order time of the next triggering time window by increasing the tolerable out-of-order time of the current triggering time window if the message efficiency is not greater than the message efficiency threshold.

[0030] Preferably, the first calculation module is specifically used to:

[0031] Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window; calculate the message efficiency U up to the current time according to the following formula n :

[0032]

[0033] Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, D ui The number of invalid messages within the calculation time of the i-th trigger time window in history.

[0034] Preferably, the second calculation module is specifically used for:

[0035] Reduce the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0036]

[0037] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and ceil() means rounding up.

[0038] Preferably, the third calculation module is specifically used for:

[0039] Increase the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0040]

[0041] Among them, n is the number of times the time window is triggered, OTn+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and floor() means rounding down.

[0042] The present invention provides a dynamic window adjustment method and device, which can dynamically adjust the tolerable disorder time according to the message efficiency threshold set by the user each time the time window is triggered for real-time calculation. This can adapt to the tolerable disorder time of different application scenarios, thereby achieving the purpose of dynamically adjusting the time window trigger time, completing the statistical analysis of the time window in a timely manner, and improving the timeliness and accuracy of the real-time calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A method flow chart of a dynamic window adjustment method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the structure of a dynamic window adjustment device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0048] To facilitate understanding of this application, first introduce the relevant concepts:

[0049] Message efficiency: the ratio of the number of messages read in that are not discarded due to message disorder to the number of messages read in;

[0050] Message efficiency threshold: During real-time calculation, it may not be possible to ensure that every message is counted due to message disorder. The business side can set a message efficiency threshold to ensure the proportion of messages used for statistics in the read messages, thereby ensuring the validity of the statistical results and reducing errors.

[0051] Tolerable out-of-order time: During real-time computing, out-of-order messages may cause the corresponding messages to not be read in during the statistical analysis time window. Setting the tolerable out-of-order time through the watermark mechanism can alleviate the amount of messages discarded due to out-of-order messages to a certain extent.

[0052] The embodiment of the present invention provides a dynamic window adjustment method, the method flow chart of which is as follows: Figure 1 As shown, the following steps are included:

[0053] S10, when performing real-time calculation in this trigger time window, calculate the message efficiency up to the current time.

[0054] In an embodiment of the present invention, during the real-time calculation of each trigger time window, the ratio of the number of messages that have not been discarded (i.e., valid messages) to the number of messages read in during the period from the first trigger time window to the current trigger time window can be counted, i.e., the message efficiency.

[0055] Of course, in actual applications, the number of messages read in and the number of messages discarded (i.e., invalid messages) can be counted by registering accumulators V and U. Each time a message is read in, the accumulator V increases by 1, and each time a message is discarded due to delay (the message timestamp is less than the watermark time corresponding to the time window), the accumulator U increases by 1. Each time the time window is triggered, the values ​​of the accumulators V and U within the calculation time are recorded and reset to 0.

[0056] In the specific implementation process, the "calculating the message efficiency up to the current time" in step S10 can be performed using the following steps:

[0057] Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window;

[0058] Calculate the message efficiency U up to the current time according to the following formula (1): n :

[0059]

[0060] Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, Dui The number of invalid messages within the calculation time of the i-th trigger time window in history.

[0061] In the embodiment of the present invention, the number of messages read in and the number of invalid messages within the calculation time of each triggering time window are obtained by reading the values ​​of the accumulators V and U recorded in each triggering time window.

[0062] It should be noted that the above embodiment uses the records up to the current time window of each trigger. In actual application, the records of one or more trigger time windows may be specified to calculate the message efficiency, which is not limited in this embodiment.

[0063] S20, determine whether the message efficiency is greater than a preset message efficiency threshold; if so, execute step S30; if not, execute step S40.

[0064] S30, obtaining the tolerable out-of-order time of the next triggering time window by reducing the tolerable out-of-order time of the current triggering time window.

[0065] In the embodiment of the present invention, the tolerable disorder time of the current trigger time window may be reduced according to a preset reduction rule, such as reducing a fixed value, and this embodiment does not limit this.

[0066] In the specific implementation process, since the message efficiency is greater than the message efficiency threshold, it indicates that the current message efficiency meets the user demand, the tolerable disorder time can be appropriately reduced to speed up the statistical analysis of the trigger time window.

[0067] Step S30 "obtaining the tolerable out-of-order time of the next trigger time window by reducing the tolerable out-of-order time of the current trigger time window" can adopt the following steps. When the message efficiency is greatly different from the message efficiency threshold, the adjustment range of the tolerable out-of-order time is also large. Conversely, when the message efficiency is less different from the message efficiency threshold, the adjustment range of the tolerable out-of-order time is also small:

[0068] According to the following formula (2), the tolerable out-of-order time OT of this trigger time window is reduced. n :

[0069]

[0070] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and ceil() means rounding up.

[0071] S40, obtaining the tolerable out-of-order time of the next triggering time window by increasing the tolerable out-of-order time of the current triggering time window.

[0072] In the embodiment of the present invention, the tolerable disorder time of the current trigger time window may be increased according to a preset increase rule, such as increasing a fixed value, and this embodiment does not limit this.

[0073] In the specific implementation process, since the message efficiency is less than the message efficiency threshold, it indicates that the current message efficiency does not meet user needs. It is necessary to appropriately increase the tolerable disorder time to reduce the amount of message discards caused by delays.

[0074] Step S40 "obtaining the tolerable out-of-order time of the next trigger time window by increasing the tolerable out-of-order time of the current trigger time window" can adopt the following steps. When the message efficiency is greatly different from the message efficiency threshold, the adjustment range of the tolerable out-of-order time is also large. Conversely, when the message efficiency is less different from the message efficiency threshold, the adjustment range of the tolerable out-of-order time is also small:

[0075] According to the following formula (3), the tolerable out-of-order time OT of this trigger time window is increased. n :

[0076]

[0077] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the minimum value, and floor() means rounding down.

[0078] The dynamic window adjustment method provided by the embodiment of the present invention can dynamically adjust the tolerable disorder time according to the message efficiency threshold set by the user each time the time window is triggered for real-time calculation. This can adapt to the tolerable disorder time of different application scenarios, thereby achieving the purpose of dynamically adjusting the time window trigger time, completing the statistical analysis of the time window in a timely manner, and improving the timeliness and accuracy of the real-time calculation.

[0079] Based on the dynamic window adjustment method provided in the above embodiment, an embodiment of the present invention provides a device for executing the above dynamic window adjustment method. The structural diagram of the device is shown in FIG. Figure 2 As shown, including:

[0080] The first calculation module 10 is used to calculate the message efficiency up to the current time when performing real-time calculation in this trigger time window;

[0081] A judging module 20, used to judge whether the message efficiency is greater than a preset message efficiency threshold;

[0082] A second calculation module 30 is used to obtain the tolerable out-of-order time of the next trigger time window by reducing the tolerable out-of-order time of the current trigger time window if the message efficiency is greater than the message efficiency threshold;

[0083] The third calculation module 40 is used to obtain the tolerable out-of-order time of the next trigger time window by increasing the tolerable out-of-order time of the current trigger time window if the message efficiency is not greater than the message efficiency threshold.

[0084] Optionally, the first calculation module 10 is specifically configured to:

[0085] Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window; calculate the message efficiency U up to the current time according to the following formula n :

[0086]

[0087] Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, D ui The number of invalid messages within the calculation time of the i-th trigger time window in history.

[0088] Optionally, the second calculation module 30 is specifically configured to:

[0089] Reduce the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0090]

[0091] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and ceil() means rounding up.

[0092] Optionally, the third calculation module 40 is specifically configured to:

[0093] Increase the tolerable out-of-order time OT of this trigger time window according to the following formula: n :

[0094]

[0095] Among them, n is the number of times the time window is triggered, OT n+1 It is the tolerable disorder time of the next trigger time window. max() means taking the maximum value, and floor() means rounding down.

[0096] The dynamic window adjustment device provided by the embodiment of the present invention can dynamically adjust the tolerable disorder time according to the message efficiency threshold set by the user each time the time window is triggered for real-time calculation. This can adapt to the tolerable disorder time of different application scenarios, thereby achieving the purpose of dynamically adjusting the time window trigger time, completing the statistical analysis of the time window in a timely manner, and improving the timeliness and accuracy of the real-time calculation.

[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0100] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A dynamic window adjustment method, characterized in that: The method comprises: When performing real-time calculations in this trigger time window, the efficiency of the message up to the current time is calculated; Determining whether the message effectiveness is greater than a preset message effectiveness threshold; If the message efficiency is greater than the message efficiency threshold, the tolerable out-of-order time of the next trigger time window is obtained by reducing the tolerable out-of-order time of the current trigger time window; If the message efficiency is not greater than the message efficiency threshold, the tolerable out-of-order time of the next triggering time window is obtained by increasing the tolerable out-of-order time of the current triggering time window.

2. The method according to claim 1, characterized in that The calculation of the message efficiency up to the present time includes: Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window; The message efficiency U up to the current time is calculated according to the following formula n : Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, D ui The number of invalid messages within the calculation time of the i-th trigger time window in history.

3. The method according to claim 1, characterized in that The method of obtaining the tolerable out-of-order time of the next triggering time window by reducing the tolerable out-of-order time of the current triggering time window includes: Reduce the tolerable out-of-order time OT of this trigger time window according to the following formula: n : Among them, n is the number of times the time window is triggered, OT n+1 is the tolerable disorder time of the next trigger time window, max() means taking the maximum value, ceil() means rounding up, and T means the message efficiency threshold.

4. The method according to claim 1, characterized in that: The method of obtaining the tolerable disorder time of the next trigger time window by increasing the tolerable disorder time of the current trigger time window includes: Increase the tolerable out-of-order time OT of this trigger time window according to the following formula: n : Among them, n is the number of times the time window is triggered, OT n+1 is the tolerable disorder time of the next trigger time window, max() means taking the maximum value, floor() means rounding down, and T means the message efficiency threshold.

5. A dynamic window adjustment device, characterized in that: The device comprises: The first calculation module is used to calculate the message efficiency up to the current time when performing real-time calculation in this trigger time window; A judging module, used to judge whether the message efficiency is greater than a preset message efficiency threshold; A second calculation module is used for obtaining the tolerable out-of-order time of the next triggering time window by reducing the tolerable out-of-order time of the current triggering time window if the message efficiency is greater than the message efficiency threshold; The third calculation module is used to obtain the tolerable out-of-order time of the next triggering time window by increasing the tolerable out-of-order time of the current triggering time window if the message efficiency is not greater than the message efficiency threshold.

6. The device according to claim 5, characterized in that The first calculation module is specifically used for: Get the number of messages read and the number of invalid messages within the calculation time up to the current trigger time window; calculate the message efficiency U up to the current time according to the following formula n : Where n is the number of times the time window is triggered, D vi is the number of messages read in the calculation time of the i-th trigger window in history, D ui The number of invalid messages within the calculation time of the i-th trigger time window in history.

7. The device according to claim 5, characterized in that The second calculation module is specifically used for: Reduce the tolerable out-of-order time OT of this trigger time window according to the following formula: n : Among them, n is the number of times the time window is triggered, OT n+1 is the tolerable disorder time of the next trigger time window, max() means taking the maximum value, ceil() means rounding up, and T means the message efficiency threshold.

8. The device according to claim 5, characterized in that The third calculation module is specifically used for: Increase the tolerable out-of-order time OT of this trigger time window according to the following formula: n : Among them, n is the number of times the time window is triggered, OT n+1 is the tolerable disorder time of the next trigger time window, max() means taking the maximum value, floor() means rounding down, and T means the message efficiency threshold.

Images