Water level line self-adaptive generation method, device and equipment and storage medium

Abstract

The invention discloses a water level line self-adaptive generation method, device and equipment and a storage medium. The method comprises the steps that parameters are preset, and a micro-cluster queue and the value of a current delay period are initialized; obtaining the data volume in the micro-cluster queue, and performing the following logical judgment according to the data volume and the length of the micro-cluster queue: if the data volume is smaller than the length, obtaining the event time of the real-time data, adding the event time into the micro-cluster queue, and sending out the minimum event time in the micro-cluster queue as a water line; if the data size is equal to the length, whether the current event time is replaced or not is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period and serves as a water level line to be sent out; if the data volume is greater than the length, sending out the minimum event time in the micro-cluster queue as a water level line; and returning to the step of acquiring the data volume, and circulating the logic judgment process until the stream processing is finished. By adopting the embodiment of the invention, the requirements of real-time performance and calculation accuracy can be met while the self-adaptive generation of the water level line is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for adaptively generating a water level line. Background Art

[0002] Watermark is a solution proposed by the open source stream processing framework Apache Flink to deal with data disorder and delay caused by network delay or differences between different computing nodes. Users can set the waiting time for delayed data to arrive according to the degree of data disorder. When the set delay time is over, the system will assume that all delayed data has arrived and immediately trigger the calculation operation.

[0003] In the open source environment, the event-time-based watermark generation strategy includes a periodic watermark generation strategy, that is, inserting a watermark into the data stream at a fixed time interval. Figure 1 As shown, Figure 1 The relationship between the system time data and the real-time number data is read every 5 minutes. Figure 1 It can be seen that at the 4th minute, there are data of both period 1 and period 2. If the watermark update strategy is to use the maximum event time as the watermark, the data of period 1 that arrives 4 minutes later will be lost. If the watermark is set back for a period of time based on the original event time, for example, the current maximum event time is maxTs, and the maximum allowed delay time is delayTime, then the watermark issued is maxTs-delayTime. Since the event time of periodic data increases in a periodic step-by-step manner rather than continuously, no matter how delayTime is set, the delay time will at least exceed one period, making it difficult to meet real-time requirements.

[0004] In addition, the prior art also collects data streams to form microclusters, calculates the degree of disorder based on the microcluster information, and dynamically adjusts the watermark based on the currently calculated degree of disorder. This method takes into account the impact of disorder on data delay, and can generate and adjust the watermark based on the degree of delay. However, the distribution of disorder is not always uniform. When the disorder of the collected microclusters is low, the calculated watermark delay is small, but the disorder of the subsequent microclusters is very high, and the calculated watermark delay is larger than the previous one. Since the watermark has the characteristic of monotonically increasing, the system can only issue watermarks with larger delays, which may result in the loss of a large amount of delayed data. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for adaptively generating a water level line, so as to solve the problems of losing a large amount of delayed data and difficulty in meeting real-time requirements in the existing water level line generation technology, and meet the requirements of real-time requirements and calculation accuracy.

[0006] To achieve the above object, an embodiment of the present invention provides a method for adaptively generating a water level line, comprising:

[0007] Preset the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period, and initialize the values ​​of the micro-cluster queue and the current delay period;

[0008] The amount of data in the micro-cluster queue is obtained, and the following logical judgment is performed according to the amount of data and the length of the micro-cluster queue:

[0009] If the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark;

[0010] If the data volume is equal to the length, determining whether to replace the current event time and send it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0011] If the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark;

[0012] Return to the step of obtaining the amount of data in the micro-cluster queue, and loop the logic judgment process until the flow processing is completed.

[0013] As an improvement of the above solution, judging whether to replace the current event time and issue it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period includes:

[0014] If the event time of a first number of data in the micro-cluster queue is greater than the current watermark, the first number of event times are selected from the micro-cluster queue according to a preset order to replace the current event time, and are sent as the watermark;

[0015] If the event time of a second number of data in the micro-cluster queue is greater than the current watermark and the delay period is greater than the maximum delay period, the second number of event times are selected from the micro-cluster queue according to the preset order to replace the current event time, and are issued as the watermark;

[0016] If there is no data of the second number in the micro-cluster queue whose event time is greater than the current watermark, the current delay period is cumulatively increased by one, and the current event time is issued as the watermark.

[0017] As an improvement of the above solution, the first number is the product of the length of the micro-cluster queue and the required maximum accuracy.

[0018] As an improvement of the above solution, the second number is the product of the length of the micro-cluster queue and the required minimum accuracy.

[0019] As an improvement of the above solution, the preset sorting is sorting from small to large.

[0020] As an improvement of the above solution, the required maximum accuracy is preset to 100%, indicating that all data in the micro-cluster queue are data of the next cycle;

[0021] The required minimum accuracy is preset to 98%, indicating that a third amount of data loss is allowed at most in a micro-cluster queue;

[0022] The maximum delay period is preset to 0, indicating that the watermark will be issued regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy;

[0023] The length of the micro-cluster queue is set according to the amount of data in each cycle of the measurement data, and generally does not exceed the amount of data in one cycle.

[0024] As an improvement of the above solution, the third number is the product of the length of the micro-cluster queue and the difference; wherein the difference is the difference between 1 and the required minimum accuracy.

[0025] To achieve the above object, an embodiment of the present invention further provides a water level adaptive generation device, comprising:

[0026] A parameter setting module is used to pre-set the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period, and initialize the value of the micro-cluster queue and the current delay period;

[0027] The waterline generation module is used to obtain the amount of data in the micro-cluster queue and perform the following logical judgment according to the amount of data and the length of the micro-cluster queue:

[0028] If the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark;

[0029] If the data volume is equal to the length, determining whether to replace the current event time and send it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0030] If the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark;

[0031] The step loop module is used to return to the step of obtaining the amount of data in the micro-cluster queue and loop the logic judgment process until the flow processing is completed.

[0032] To achieve the above objectives, an embodiment of the present invention further provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the above water level adaptive generation method when executing the computer program.

[0033] To achieve the above objectives, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-mentioned water level adaptive generation method.

[0034] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer program product, which includes a computer program; wherein, when the computer program is running, it controls the device where the program product is located to execute the above-mentioned water level adaptive generation method.

[0035] Implementing the embodiments of the present invention has the following beneficial effects:

[0036] The method, device, equipment and storage medium for adaptively generating a watermark provided by the embodiments of the present invention initialize the microcluster queue and the value of the current delay period by presetting the length of the microcluster queue, the maximum accuracy required, the minimum accuracy required, the watermark sending period and the maximum delay period; obtain the amount of data in the microcluster queue, and perform the following logical judgment based on the amount of data and the length of the microcluster queue: if the amount of data is less than the length, obtain the event time of the real-time data and add it to the microcluster queue, and send the minimum event time in the microcluster queue as the watermark; if the amount of data is equal to the length, then According to the required maximum accuracy, the required minimum accuracy and the maximum delay period, determine whether to replace the current event time and issue it as a watermark; if the data volume is greater than the length, issue the minimum event time in the micro-cluster queue as the watermark; return to the step of obtaining the data volume in the micro-cluster queue, loop the logic judgment process until the stream processing is completed, and can process the delayed data in the real-time data stream by flexibly configuring the required maximum accuracy and the required minimum accuracy, realize the adaptive dynamic generation of the watermark, thereby reducing the loss of delayed data, shortening the delay time, and meeting the requirements of real-time and calculation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the relationship between the system time data and the real-time number data read by Flink in a cycle of every 5 minutes provided by an embodiment of the present invention;

[0038] Figure 2 is a flow chart of a method for adaptively generating a water level line provided by an embodiment of the present invention;

[0039] Figure 3 is a state diagram of micro-cluster data provided by an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the number of data lost in different periods of a day in a normal water level line provided by an embodiment of the present invention compared with the water level line adaptive method of the present invention;

[0041] Figure 5 It is a schematic diagram of the time of delaying the output of the water level line by one day compared with the normal water level line by using the water level line adaptive method of the present invention provided by the embodiment of the present invention;

[0042] Figure 6 is a structural block diagram of a water level adaptive generation device provided by an embodiment of the present invention;

[0043] Figure 7 It is a structural block diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be noted that the step numbers in the text are only for the convenience of explanation of specific embodiments and do not serve to limit the order in which the steps are executed. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0046] See also Figure 2 , Figure 2 : is a flow chart of a method for adaptively generating a water level line provided by an embodiment of the present invention, wherein the method for adaptively generating a water level line comprises steps S1 to S2, which are specifically as follows:

[0047] S1, pre-setting the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period parameters, and initializing the value of the micro-cluster queue and the current delay period;

[0048] S2. Obtain the amount of data in the micro-cluster queue, and perform the following logical judgment based on the amount of data and the length of the micro-cluster queue:

[0049] S21, if the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark;

[0050] S22, if the data volume is equal to the length, determine whether to replace the current event time and issue it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0051] S23, if the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark;

[0052] S3, returning to the step of obtaining the amount of data in the micro-cluster queue, and looping the logic judgment process until the flow processing is completed.

[0053] It can be understood that the micro-cluster refers to a data structure that stores the event time of data in a queue of fixed length according to the order in which the data arrives.

[0054] In the embodiment of the present invention, the accuracy of watermark generation is guaranteed by presetting the length of the micro-cluster, the required maximum accuracy, the required minimum accuracy and the maximum tolerable delay time period; if the micro-cluster queue is not full, the real-time data event time is added to the queue, and the minimum event time is taken as the watermark to be issued; if the micro-cluster queue is full, it is determined whether to replace the current event time and issue it as the watermark based on the required maximum accuracy, the required minimum accuracy and the maximum delay period, otherwise the minimum event time in the micro-cluster queue is issued as the watermark, and the logical judgment process is repeated until the stream processing is completed. The delayed data in the real-time data stream can be processed by flexibly configuring the required maximum accuracy and the required minimum accuracy, so as to realize the adaptive dynamic generation of the watermark, thereby reducing the loss of delayed data, shortening the delay time, and meeting the requirements of real-time performance and calculation accuracy.

[0055] In an optional embodiment, judging whether to replace the current event time and issue it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period includes:

[0056] If the event time of a first number of data in the micro-cluster queue is greater than the current watermark, the first number of event times are selected from the micro-cluster queue according to a preset order to replace the current event time, and are sent as the watermark;

[0057] If the event time of a second number of data in the micro-cluster queue is greater than the current watermark and the delay period is greater than the maximum delay period, the second number of event times are selected from the micro-cluster queue according to the preset order to replace the current event time, and are issued as the watermark;

[0058] If there is no data of the second number in the micro-cluster queue whose event time is greater than the current watermark, the current delay period is cumulatively increased by one, and the current event time is issued as the watermark.

[0059] In the embodiment of the present invention, a certain amount of data is collected through a micro-cluster, and the percentage of tolerable data loss is calculated based on the preset amount. The current event time is re-determined in combination with the delay period and issued as a watermark, ensuring that the watermark can be dynamically adjusted with changes in time and data status. By setting the maximum delay period, it is ensured that the system will not affect real-time processing due to long-term delayed data.

[0060] In an optional embodiment, the first number is the product of the length of the micro-cluster queue and the required maximum accuracy.

[0061] In an optional embodiment, the second number is the product of the length of the micro-cluster queue and the required minimum accuracy.

[0062] In an embodiment of the present invention, based on the characteristics that the amount of periodic data is stable between cycles, and the amount of delayed data in the same cycle decreases rapidly as time increases, a method for adaptively generating a watermark is proposed. The accuracy of the real-time calculation is determined according to the pre-set length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, and the maximum tolerated delay period. The length of the micro-cluster queue is the number of data that can be cached at one time. The longer the length of the micro-cluster queue, the more accurate the generated watermark, and the less likely it is to lose data. Therefore, the required maximum accuracy is the percentage of the micro-cluster queue that is greater than the current watermark. If the event time that meets the required maximum accuracy is greater than the current watermark, the minimum value of the event time that meets the required maximum accuracy can be issued as the watermark, and the l*αth (where l is the length of the micro-cluster queue and α is the required maximum accuracy) event time after sorting is taken as the current event time. At this stage, the data of the next cycle that occasionally appears is filtered out, such as Figure 2As shown, there are 30 events in the micro-cluster queue, and only 5 new events of the next cycle arrive. If the required minimum accuracy is 90%, at least 27 events with event time 5 in the micro-cluster queue must arrive before the water level 5 is issued to the downstream.

[0063] In an optional embodiment, the preset sorting is sorting from small to large.

[0064] In an optional embodiment, the required maximum accuracy is preset to 100%, indicating that all data in the micro-cluster queue are data of the next cycle;

[0065] The required minimum accuracy is preset to 98%, indicating that a third amount of data loss is allowed at most in a micro-cluster queue;

[0066] The maximum delay period is preset to 0, indicating that the watermark will be issued regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy;

[0067] The length of the micro-cluster queue is set according to the amount of data in each cycle of the measurement data, and generally does not exceed the amount of data in one cycle.

[0068] It can be understood that the length of the micro-cluster queue is the number of data that can be cached at one time. The longer the length of the micro-cluster queue, the more accurate the generated watermark is and the smaller the possibility of data loss is. Therefore, the maximum accuracy required is the percentage in the micro-cluster queue that is greater than the current watermark; the minimum accuracy required is used to determine whether there is data for the next cycle.

[0069] In an optional embodiment, the third number is the product of the length of the micro-cluster queue and the difference value; wherein the difference value is the difference between 1 and the required minimum accuracy.

[0070] The method for adaptively generating a watermark provided by an embodiment of the present invention initializes the value of the micro-cluster queue and the current delay period by presetting the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period, and the maximum delay period; obtains the amount of data in the micro-cluster queue, and performs the following logical judgment according to the amount of data and the length of the micro-cluster queue: if the amount of data is less than the length, obtains the event time of the real-time data and adds it to the micro-cluster queue, and sends the minimum event time in the micro-cluster queue as the watermark; if the amount of data is equal to the length, determines whether to replace the current event time and send it as the watermark according to the required maximum accuracy, the required minimum accuracy, and the maximum delay period; if the amount of data is greater than the length, sends the minimum event time in the micro-cluster queue as the watermark; returns to the step of obtaining the amount of data in the micro-cluster queue, loops the logical judgment process, and ends the stream processing. The method can process the delayed data in the real-time data stream by flexibly configuring the required maximum accuracy and the required minimum accuracy, and realizes the adaptive dynamic generation of the watermark, thereby reducing the loss of delayed data, shortening the delay time, and meeting the requirements of real-time performance and calculation accuracy.

[0071] Exemplarily, the adaptive waterline generation method includes steps 1 to 4:

[0072] Step 1: Set the watermark sending period. The maximum accuracy α is 100% by default, the minimum accuracy β is 98% by default, the maximum delay period m is 0 by default, the micro-cluster queue timestampqueue with a length of l is initialized, and the current delay period delay_period is initialized to 0.

[0073] Step 2: If the data size in the timestampqueue is greater than l, the event time eventtime of the real-time data is taken out and added to the timestampqueue, and then the smallest eventtime in the timestampqueue is taken as the watermark to be emitted;

[0074] Step 3: If the amount of data in timestampqueue is equal to l, then:

[0075] Step 3.1: Determine whether there are l*α data in timestampqueue whose eventtime is greater than the current watermark. If so, take out the l*αth event time sorted from small to large and replace the current event time currenteventtime, and emit it as the watermark; queueeventtime α

[0076] Step 3.2: Determine whether there are l*β data in timestampqueue whose eventtime is greater than the current watermark and whose delay period is greater than m. If so, take out the l*βth event time sorted from small to large to replace the current event time currenteventtime and emit it as the watermark; queueeventtime β

[0077] Step 3.3: If there are no l*β data in the timestampqueue whose eventtime is greater than the current watermark, delay_period is incremented by 1, and currenteventtime is emitted as the watermark.

[0078] Step 4: If the data size in the timestampqueue is less than l, the smallest eventtime in the queue is taken out and sent as the watermark.

[0079] Repeat steps 2 to 4 until the stream processing is completed.

[0080] In the specific implementation process, based on the 1.8 version of the Java library, the watermark generation mechanism of Flink can be rewritten in Java language according to the solution of the present invention, and referenced when starting the job.

[0081] The present invention starts the flink job after presetting the length of the micro cluster to 20000, the maximum accuracy required to be 100%, the minimum accuracy required to be 98%, and the maximum tolerable delay time period to be 1. At the same time, a flink job with only the minimum accuracy required to be 98% is started as a control group, and the statistical results of the generated watermarks and the number of captured delay data items are compared between the two jobs when the maximum tolerable delay is 1 period.

[0082] As shown in the statistical results of Table 1, under the condition of the same delay time, the accuracy of the statistical results obtained in real-time calculation using the solution of the present invention is greater than the accuracy of the statistical results calculated using the original Flink watermark generation method.

[0083] Table 1 Statistical results of normal water level and water level adaptively generated by the present invention within one day

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] The statistical result of the number of capture delay data in Table 1 is the number of lost data in different periods of a day compared with the normal water level line and the adaptive water level line of the present invention. The curve of the statistical result is as follows: Figure 4 shown.

[0092] As can be seen from Table 1, in each cycle of a day, if the watermark is issued according to the normal event time in the first cycle, 1484 (i.e., the sum of the number of delayed data captured in Table 1) events will be lost. The watermark generated by the present invention has a larger statistical result delay than the watermark generated by the original event time. Figure 5 As shown, the maximum delay time is 7 minutes, which is much shorter than the usual delay time of 15 minutes per cycle.

[0093] In summary, the required maximum accuracy and required minimum accuracy can be set according to the real-time and accuracy requirements of the job. If the real-time requirement is high, set the required minimum accuracy to a small acceptable value and set the tolerance delay period to 0; if the accuracy requirement is high, make the opposite settings, set the required minimum accuracy to a large value and set the tolerance delay period to an acceptable large value.

[0094] See also Figure 6 , Figure 6 : is a structural block diagram of a water level adaptive generation device 10 provided in an embodiment of the present invention, wherein the water level adaptive generation device 10 comprises:

[0095] The parameter setting module 11 is used to pre-set the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period, and initialize the value of the micro-cluster queue and the current delay period;

[0096] The waterline generation module 12 is used to obtain the amount of data in the micro-cluster queue and perform the following logical judgment according to the amount of data and the length of the micro-cluster queue:

[0097] If the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark;

[0098] If the data volume is equal to the length, determining whether to replace the current event time and send it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0099] If the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark;

[0100] The step loop module 13 is used to return to the step of obtaining the amount of data in the micro-cluster queue and loop the logic judgment process until the flow processing is completed.

[0101] Optionally, the water level line generating module 12 is further used for:

[0102] If the event time of a first number of data in the micro-cluster queue is greater than the current watermark, the first number of event times are selected from the micro-cluster queue according to a preset order to replace the current event time, and are sent as the watermark;

[0103] If the event time of a second number of data in the micro-cluster queue is greater than the current watermark and the delay period is greater than the maximum delay period, the second number of event times are selected from the micro-cluster queue according to the preset order to replace the current event time, and are issued as the watermark;

[0104] If there is no data of the second number in the micro-cluster queue whose event time is greater than the current watermark, the current delay period is cumulatively increased by one, and the current event time is issued as the watermark.

[0105] Optionally, the first number is the product of the length of the micro-cluster queue and the required maximum accuracy.

[0106] Optionally, the second number is the product of the length of the micro-cluster queue and the required minimum accuracy.

[0107] Optionally, the preset sorting is sorting from small to large.

[0108] Optionally, the required maximum accuracy is preset to 100%, indicating that all data in the micro-cluster queue are data of the next cycle;

[0109] The required minimum accuracy is preset to 98%, indicating that a third amount of data loss is allowed at most in a micro-cluster queue;

[0110] The maximum delay period is preset to 0, indicating that the watermark will be issued regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy;

[0111] The length of the micro-cluster queue is set according to the amount of data in each cycle of the measurement data, and generally does not exceed the amount of data in one cycle.

[0112] Optionally, the third number is the product of the length of the micro-cluster queue and a difference value; wherein the difference value is a difference between 1 and the required minimum accuracy.

[0113] It is worth noting that the working process of each module in the water level adaptive generation device 10 described in the embodiment of the present invention can refer to the working process of the water level adaptive generation method described in the above embodiment, and will not be repeated here.

[0114] The watermark adaptive generation device provided by the embodiment of the present invention initializes the value of the micro-cluster queue and the current delay period by pre-setting the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period; obtains the amount of data in the micro-cluster queue, and performs the following logical judgment according to the amount of data and the length of the micro-cluster queue: if the amount of data is less than the length, obtains the event time of the real-time data and adds it to the micro-cluster queue, and sends the minimum event time in the micro-cluster queue as the watermark; if the amount of data is equal to the length, judges whether to replace the current event time and send it as the watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; if the amount of data is greater than the length, sends the minimum event time in the micro-cluster queue as the watermark; returns to the step of obtaining the amount of data in the micro-cluster queue, loops the logical judgment process until the stream processing is completed, and can process the delayed data in the real-time data stream by flexibly configuring the required maximum accuracy and the required minimum accuracy, so as to realize the adaptive dynamic generation of the watermark, thereby reducing the loss of delayed data, shortening the delay time, and meeting the requirements of real-time performance and calculation accuracy.

[0115] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-mentioned water level adaptive generation method.

[0116] An embodiment of the present invention further provides a computer program product, which includes a computer program; wherein, when the computer program is running, the device where the computer program product is located is controlled to execute the above-mentioned water level adaptive generation method.

[0117] See also Figure 7 , Figure 7: is a structural block diagram of a computer device 20 provided in an embodiment of the present invention, and the computer device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned water level adaptive generation method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0118] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device 20.

[0119] The computer device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the computer device 20 and does not limit the computer device 20. The computer device 20 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the computer device 20 may also include input and output devices, network access devices, buses, etc.

[0120] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the computer device 20, and uses various interfaces and lines to connect various parts of the entire computer device 20.

[0121] The memory 22 can be used to store the computer program and / or module. The processor 21 implements various functions of the computer device 20 by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0122] Wherein, if the module / unit integrated in the computer device 20 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 21. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0123] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for adaptively generating a water level line, characterized in that: include: Preset the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period, and initialize the values ​​of the micro-cluster queue and the current delay period; The amount of data in the micro-cluster queue is obtained, and the following logical judgment is performed according to the amount of data and the length of the micro-cluster queue: If the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark; If the data volume is equal to the length, determining whether to replace the current event time and send it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; If the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark; Return to the step of obtaining the amount of data in the micro-cluster queue, and loop the logic judgment process until the flow processing is completed.

2. The method for adaptively generating a water level line according to claim 1, characterized in that: The step of determining whether to replace the current event time and issue it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period includes: If the event time of a first number of data in the micro-cluster queue is greater than the current watermark, the first number of event times are selected from the micro-cluster queue according to a preset order to replace the current event time, and are sent as the watermark; If the event time of a second number of data in the micro-cluster queue is greater than the current watermark and the delay period is greater than the maximum delay period, the second number of event times are selected from the micro-cluster queue according to the preset order to replace the current event time, and are issued as the watermark; If there is no data of the second number in the micro-cluster queue whose event time is greater than the current watermark, the current delay period is cumulatively increased by one, and the current event time is issued as the watermark.

3. The method for adaptively generating a water level line according to claim 2, characterized in that: The first number is a product of a length of the micro-cluster queue and the required maximum accuracy.

4. The method for adaptively generating a water level line according to claim 3, characterized in that: The second number is a product of the length of the micro-cluster queue and the required minimum accuracy.

5. The method for adaptively generating a water level line according to claim 4, characterized in that: The preset sorting is from small to large.

6. The method for adaptively generating a water level line according to claim 5, characterized in that: The required maximum accuracy is preset to 100%, indicating that all data in the micro-cluster queue are data of the next cycle; The required minimum accuracy is preset to 98%, indicating that a third amount of data loss is allowed at most in a micro-cluster queue; The maximum delay period is preset to 0, indicating that the watermark will be issued regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy; The length of the micro-cluster queue is set according to the amount of data in each cycle of the measurement data, and generally does not exceed the amount of data in one cycle.

7. The method for adaptively generating a water level line according to claim 6, characterized in that: The third number is the product of the length of the micro-cluster queue and the difference; The difference is the difference between 1 and the required minimum accuracy.

8. A water level adaptive generation device, characterized in that: include: A parameter setting module is used to pre-set the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the watermark sending period and the maximum delay period, and initialize the value of the micro-cluster queue and the current delay period; The waterline generation module is used to obtain the amount of data in the micro-cluster queue and perform the following logical judgment according to the amount of data and the length of the micro-cluster queue: If the data volume is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the minimum event time in the micro-cluster queue is sent as the watermark; If the data volume is equal to the length, determining whether to replace the current event time and send it as a watermark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; If the data volume is greater than the length, the minimum event time in the micro-cluster queue is sent as the watermark; The step loop module is used to return to the step of obtaining the amount of data in the micro-cluster queue and loop the logic judgment process until the flow processing is completed.

9. A computer device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for adaptively generating a water level line according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the method for adaptively generating a water level line according to any one of claims 1 to 7.

Images