A water level line adaptive generation method, device, equipment and storage medium

Abstract

The application discloses a water line self-adaptive generation method, device and equipment and a storage medium. The method comprises the following steps: setting parameters in advance, initializing the value of a micro-cluster queue and a current delay period; obtaining the data amount in the micro-cluster queue, and performing the following logical judgment according to the data amount and the length of the micro-cluster queue: if the data amount is less than the length, the event time of real-time data is obtained and added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as a water line and sent out; if the data amount is equal to the length, whether the current event time is replaced and taken as a water line and sent out is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; if the data amount is greater than the length, the smallest event time in the micro-cluster queue is taken as a water line and sent out; the step of obtaining the data amount is returned, and the logical judgment process is circularly performed until the stream processing is finished. By adopting the embodiment of the application, the water line self-adaptive generation can be realized while meeting the requirements of real-time performance and calculation accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a water line adaptive generation method and device, equipment and storage medium. BACKGROUND

[0002] Watermark is a solution proposed by Apache Flink, an open source stream processing framework, to deal with data out-of-order and delay caused by network delay or differences between different computing nodes. Users can set the waiting time for allowing delayed data to arrive according to the degree of data out-of-order. When the set delay time ends, the system will consider that all delayed data has arrived and will trigger the computing operation immediately.

[0003] The watermark generation strategy based on event time in the open source environment includes a periodic watermark generation strategy, that is, a watermark is inserted into the data stream every fixed time interval. As shown in Figure 1 , Figure 1 is the relationship between system time data and real-time number data with a period of 5 minutes. As can be seen from Figure 1 , at the 4th minute, there are data of period 1 and period 2 at the same time, if the watermark update strategy is to take the maximum event time as the watermark, then the data of period 1 arriving after 4 minutes will be lost. If the watermark is based on the original event time and is delayed for a period of time, such as the current maximum event time is maxTs, and the maximum delay time is delayTime, then the watermark issued is maxTs-delayTime, since the event time of the periodic data is increased in a step-by-step manner, rather than continuously, it is difficult to meet the real-time requirement regardless of how delayTime is set, the delay time will at least exceed a period.

[0004] In addition, the prior art also forms micro-clusters by collecting data streams, calculates the out-of-order degree according to the micro-cluster information, and dynamically adjusts the watermark according to the current calculated out-of-order degree. This method takes into account the influence of out-of-order degree on data delay, and can generate and adjust the watermark according to the delay degree. However, the distribution of out-of-order degree is not always uniform. When the collected micro-cluster out-of-order degree is low, the calculated watermark delay is small, but the out-of-order degree of the subsequent micro-cluster is high, and the calculated watermark delay is larger than before. Since the watermark has the characteristic of monotonic increase, the system can only issue a watermark with large delay, which may result in the loss of a large amount of delayed data. SUMMARY

[0005] The application provides a water level line adaptive generation method, device, equipment and storage medium, to solve the problems of losing a large amount of delay data and being difficult to meet real-time in the existing water level line generation technology, and meet the requirements of real-time and calculation accuracy.

[0006] To achieve the above object, the application provides a water level line adaptive generation method, comprising:

[0007] The length, required maximum accuracy, required minimum accuracy, water level line sending period and maximum delay period of the micro-cluster queue are set in advance, and the values of the micro-cluster queue and the current delay period are initialized;

[0008] The data amount in the micro-cluster queue is obtained, and the following logical judgment is performed according to the data amount and the length of the micro-cluster queue:

[0009] If the data amount is less than the length, the event time of real-time data is added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as the water level line to be sent out;

[0010] If the data amount is equal to the length, whether to replace the current event time and take it as the water level line to be sent out is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0011] If the data amount is greater than the length, the smallest event time in the micro-cluster queue is taken as the water level line to be sent out;

[0012] The step of obtaining the data amount in the micro-cluster queue is returned, and the logical judgment process is circularly performed until the stream processing is completed.

[0013] As an improvement of the above scheme, whether to replace the current event time and take it as the water level line to be sent out according to the required maximum accuracy, the required minimum accuracy and the maximum delay period comprises:

[0014] If the event time of the data with the first number of quantities in the micro-cluster queue is greater than the current water level line, the first number of event times are selected in the micro-cluster queue according to the preset sorting to replace the current event time, and the water level line is sent out;

[0015] If the event time of the data with the second number of quantities in the micro-cluster queue is greater than the current water level line and the delay period is greater than the maximum delay period, the second number of event times are selected in the micro-cluster queue according to the preset sorting to replace the current event time, and the water level line is sent out;

[0016] If the event time of the data with the second number of quantities in the micro-cluster queue is not greater than the current water level line, the current delay period is accumulated by one, and the current event time is taken as the water level line to be sent out.

[0017] As an improvement of the above scheme, the first number is the product of the length of the micro-cluster queue and the required maximum accuracy.

[0018] As an improvement of the above scheme, the second number is the product of the length of the micro-cluster queue and the required minimum accuracy.

[0019] As an improvement of the above scheme, the preset order is from small to large.

[0020] As an improvement of the above scheme, the required maximum accuracy is preset as 100%, indicating that all data in the micro-cluster queue is the data of the next period;

[0021] The required minimum accuracy is preset as 98%, indicating that the maximum number of data loss allowed in a micro-cluster queue is the third number;

[0022] The maximum delay period is preset as 0, indicating that the water line will be sent regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy.

[0023] The length of the micro-cluster queue is set according to the data size of each period in the measurement data, and generally does not exceed the data size of one period.

[0024] As an improvement of the above scheme, the third number is the product of the length of the micro-cluster queue and the difference; wherein the difference is the difference between 1 and the required minimum accuracy.

[0025] To achieve the above object, the embodiment of the present application further provides a water line self-adaptive generation device, comprising:

[0026] A parameter setting module is configured to preset the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the water line sending period and the maximum delay period, and initialize the values of the micro-cluster queue and the current delay period.

[0027] A water line generation module is configured to obtain the data size in the micro-cluster queue, and perform the following logical judgment according to the data size and the length of the micro-cluster queue:

[0028] If the data size is less than the length, the event time of the real-time data is added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as the water line to be sent.

[0029] If the data size is equal to the length, it is judged whether to replace the current event time and take it as the water line to be sent according to the required maximum accuracy, the required minimum accuracy and the maximum delay period.

[0030] If the data amount is greater than the length, the smallest event time in the micro-cluster queue is sent as the water line;

[0031] The step cycle module is configured to return to the step of obtaining the data amount in the micro-cluster queue, and the cycle logic judgment process is repeated until the stream processing is completed.

[0032] To achieve the above object, the embodiment of the present application further provides a computer device, which comprises a processor, a memory and a computer program stored in the memory and arranged to be executed by the processor, and the processor realizes the water line adaptive generation method when executing the computer program.

[0033] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the water line adaptive generation method when running.

[0034] To achieve the above object, the embodiment of the present application further provides a computer program product, which comprises a computer program; wherein the computer program controls the device where the program product is located to execute the water line adaptive generation method when running.

[0035] The embodiment of the present application has the following beneficial effects:

[0036] The water line adaptive generation method, device, equipment and storage medium provided by the embodiment of the present application can initialize the values of the micro-cluster queue and the current delay period by pre-setting the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the water line sending period and the maximum delay period; obtain the data amount in the micro-cluster queue, and perform the following logical judgment according to the data amount and the length of the micro-cluster queue: if the data amount is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is sent as the water line; if the data amount is equal to the length, whether to replace the current event time and send it as the water line is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; if the data amount is greater than the length, the smallest event time in the micro-cluster queue is sent as the water line; the step of obtaining the data amount in the micro-cluster queue is returned, and the cycle logical judgment process is repeated until the stream processing is completed. The delay data in the real-time data stream can be processed by flexibly configuring the required maximum accuracy and the required minimum accuracy, the adaptive dynamic generation of the water line is realized, the loss of delay data is reduced, the delay time is shortened, and the requirements of real-time and calculation accuracy are met. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of flink provided by the embodiment of the present application, which reads the relationship between system time data and real-time number data every 5 minutes;

[0038] Figure 2 is a flow chart of the water line adaptive generation method provided by the embodiment of the present application;

[0039] Figure 3 is a state diagram of micro-cluster data provided by the embodiment of the present application;

[0040] Figure 4 is a schematic diagram of the number of data lost in different periods of a day compared with the water line adaptive method provided by the embodiment of the present application;

[0041] Figure 5 is a schematic diagram of the time delayed in a day compared with the normal water line adaptive method provided by the embodiment of the present application;

[0042] Figure 6 is a structural block diagram of the water line adaptive generation device provided by the embodiment of the present application;

[0043] Figure 7 is a structural block diagram of the computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] In the description of the present application, it should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments, and do not serve as the function of limiting the execution sequence of the steps. The terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0046] Referring to Figure 2 , Figure 2 is a flow chart of the water line adaptive generation method provided by the embodiment of the present application, which includes steps S1 to S2, and the details are as follows:

[0047] S1, parameters of length of micro-cluster queue, required maximum accuracy, required minimum accuracy, water mark sending period and maximum delay period are set in advance, and values of the micro-cluster queue and the current delay period are initialized;

[0048] S2, an amount of data in the micro-cluster queue is obtained, and the following logical judgment is made according to the amount of data and the length of the micro-cluster queue:

[0049] S21, if the amount of data is less than the length, an event time of real-time data is added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is sent as a water mark;

[0050] S22, if the amount of data is equal to the length, whether to replace the current event time and send it as a water mark is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0051] S23, if the amount of data is greater than the length, the smallest event time in the micro-cluster queue is sent as a water mark;

[0052] S3, the step of obtaining the amount of data in the micro-cluster queue is returned, and the logical judgment process is repeated until the stream processing is finished.

[0053] It can be understood that the micro-cluster refers to a data structure in which event times of data are stored in a queue with a fixed length according to the order of data arrival.

[0054] In the embodiment of the application, the length of the micro-cluster, the required maximum accuracy, the required minimum accuracy and the maximum tolerance delay time period are set in advance to ensure the accuracy of the water mark generation; if the micro-cluster queue is not full, the event time of real-time data is added to the queue, and the smallest event time is taken as a water mark; if the micro-cluster queue is full, whether to replace the current event time and send it as a water mark is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period, otherwise the smallest event time in the micro-cluster queue is sent as a water mark, and the logical judgment process is repeated until the stream processing is finished, so that the delay data in the real-time data stream can be processed by flexibly configuring the required maximum accuracy and the required minimum accuracy, the water mark can be adaptively and dynamically generated, the loss of delay data is reduced, the delay time is shortened, and the requirements of real-time and calculation accuracy are met.

[0055] In an optional embodiment, whether to replace the current event time and send it as a water mark according to the required maximum accuracy, the required minimum accuracy and the maximum delay period includes:

[0056] If the event time of the data of the first number of the micro-cluster queue is greater than the current water line, the first number of the event time in the micro-cluster queue is selected according to the preset sorting to replace the current event time, and is sent as the water line;

[0057] If the event time of the data of the second number of the micro-cluster queue is greater than the current water line and the delay period is greater than the maximum delay period, the second number of the event time in the micro-cluster queue is selected according to the preset sorting to replace the current event time, and is sent as the water line;

[0058] If the event time of the data of the second number of the micro-cluster queue does not exist and is greater than the current water line, the current delay period is accumulated by one, and the current event time is sent as the water line.

[0059] In the embodiment of the application, a certain number of data are collected through the micro-cluster, and the percentage of the data that can be tolerated to be lost is calculated according to the preset number, the current event time is re-determined according to the delay period and is sent as the water line, so that the water line can be dynamically adjusted according to the change of time and data state, and the maximum delay period is set to ensure that the system will not be affected by the real-time processing due to long-delay data.

[0060] In an optional embodiment, the first number is the product of the length of the micro-cluster queue and the required maximum accuracy.

[0061] In an optional embodiment, the second number is the product of the length of the micro-cluster queue and the required minimum accuracy.

[0062] In the embodiment of the application, based on the characteristics that the data amount of the periodic data is stable in a period, and the data amount of the same period that is delayed to arrive decreases rapidly with the increase of time, a water line adaptive generation method is proposed. The accuracy of real-time calculation is determined according to the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy and the maximum delay period that can be tolerated, the length of the micro-cluster queue is the number of data that can be cached at a time, the longer the length of the micro-cluster queue is, the more accurate the generated water line is, and the less the possibility of losing data is, so the required maximum accuracy is the percentage of the event time that is greater than the current water line in the micro-cluster queue. If the event time that meets the required maximum accuracy is greater than the current water line, the minimum value of the event time that meets the required maximum accuracy is sent as the water line, and the l*αth event time (wherein l is the length of the micro-cluster queue, and α is the required maximum accuracy) is taken as the current event time. In this stage, the data of the next period that occasionally appears is filtered out, such as Figure 2As shown, there are 30 events in the micro-cluster queue, and only 5 events of the next period have arrived, if the demand minimum accuracy is 90%, at least 27 events in the micro-cluster queue are needed to reach 5 events to send the watermark 5 to the downstream.

[0063] In an alternative embodiment, the preset order is from small to large.

[0064] In an alternative embodiment, the demand maximum accuracy is preset to 100%, which means all data in the micro-cluster queue is data of the next period;

[0065] The demand minimum accuracy is preset to 98%, which means the maximum number of data loss allowed in a micro-cluster queue is the third number;

[0066] The maximum delay period is preset to 0, which means the watermark will be sent regardless of whether there is data in the micro-cluster queue that meets the demand minimum accuracy.

[0067] The length of the micro-cluster queue is set according to the amount of data in each period of the measurement data, and generally does not exceed the amount of data in one period.

[0068] It can be understood that the length of the micro-cluster queue is the number of data that can be cached at a time, the longer the length of the micro-cluster queue, the more accurate the generated watermark, and the less likely the data loss, so the demand maximum accuracy is the percentage of data in the micro-cluster queue that is greater than the current watermark; the demand minimum accuracy is used to determine whether there is data of the next period.

[0069] In an alternative embodiment, the third number is the product of the length of the micro-cluster queue and the difference; wherein the difference is the difference between 1 and the demand minimum accuracy.

[0070] The water level line adaptive generation method provided by the embodiment of the application comprises the following steps: setting the length of a micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the water level line sending period and the maximum delay period in advance, initializing the values of the micro-cluster queue and the current delay period; obtaining the data amount in the micro-cluster queue, and performing the following logical judgment according to the data amount and the length of the micro-cluster queue: if the data amount is less than the length, the event time of real-time data is obtained and added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as the water level line and sent out; if the data amount is equal to the length, whether the current event time is replaced and taken as the water level line and sent out is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; if the data amount is greater than the length, the smallest event time in the micro-cluster queue is taken as the water level line and sent out; returning to the step of obtaining the data amount in the micro-cluster queue, and performing the logical judgment process cyclically until the stream processing is completed, so that the delayed data in the real-time data stream can be processed by flexibly configuring the required maximum accuracy and the required minimum accuracy, the adaptive dynamic generation of the water level line is realized, the loss of delayed data is reduced, the delay time is shortened, and the requirements of real-time performance and calculation accuracy are met.

[0071] For example, the adaptive water level line generation method comprises steps 1 to 4.

[0072] Step 1: setting the sending period of the water level line, the required maximum accuracy α by default being 100%, the required minimum accuracy β by default being 98%, the maximum delay period m by default being 0, initializing the micro-cluster queue timestampqueue with the length l, and initializing the current delay period delay_period as 0;

[0073] Step 2: if the data amount size in the timestampqueue is greater than l, the event time eventtime of real-time data is taken out and added to the timestampqueue, and then the smallest eventtime in the timestampqueue is taken as the water level line and sent out;

[0074] Step 3: if the data amount size in the timestampqueue is equal to l, the following steps are performed:

[0075] Step 3.1: judging whether there is l*α data in the timestampqueue whose eventtime is greater than the current water level line, if there is, the l*α eventtime sorted in ascending order is taken out and used to replace the current eventtime currenteventtime, and then the current eventtime is taken as the water level line and sent out; queueeventtime α

[0076] Step 3.2, judging whether eventtime of l*β data in the timestampqueue is greater than the current waterline and delay period is greater than m, if yes, taking out the l*β eventtime in the order from small to large to replace the current eventtime currenteventtime and issuing as watermark; queueeventtime β

[0077] Step 3.3, if eventtime of l*β data in the timestampqueue is not greater than the current waterline, delay_period is accumulated by 1, and currenteventtime is issued as watermark;

[0078] Step 4, if the size of data in the timestampqueue is less than l, the smallest eventtime in the queue is taken out and issued as watermark.

[0079] The steps 2 to 4 are repeated until the stream processing is finished.

[0080] In the specific implementation process, the watermark generation mechanism of flink can be rewritten through java language according to the scheme of the application based on the 1.8 version of java library, and the implementation is referenced when starting the job.

[0081] The flink job is started after the length of the microcluster is set as 20000, the maximum required accuracy is 100%, the minimum required accuracy is 98%, and the maximum tolerant delay time period is 1. Meanwhile, a flink job only setting the minimum required accuracy as 98% is started as a control group, and the statistical results of the generated waterline and the captured delay data under the maximum tolerant delay of 1 period are compared.

[0082] As shown in the statistical results in table 1, in the case of the same delay time, the accuracy of the statistical results obtained in real-time calculation by using the scheme of the application is greater than the accuracy of the statistical results obtained by using the original flink waterline generation method.

[0083] Table 1 statistical results of normal waterline and waterline generated by the application in one day

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] The statistical result of the number of captured delay data in Table 1 is the number of lost data of the normal water line in different periods within a day compared with the adaptive water line of the present application, and the curve of the statistical result is shown in Figure 4 .

[0092] As shown in Table 1, in each period within a day, if the water line is sent according to the normal event time in the first period, 1484 (i.e. the total of the number of captured delay data in Table 1) events will be lost, and the statistical result of the output of the water line generated by the present application is more delayed than the water line generated by the original event time, as shown in Figure 5 . The maximum delay time is 7 minutes, which is much shorter than the time of 15 minutes of a normal delay period.

[0093] As shown above, the required maximum accuracy and the required minimum accuracy can be set according to the real-time and accuracy requirements of the job. If the real-time requirement is high, the required minimum accuracy is set to a small acceptable value, and the tolerance delay period is set to 0; if the accuracy requirement is high, the opposite setting is performed, the required minimum accuracy is set to a large value, and the tolerance delay period is set to an acceptable large value.

[0094] Referring to Figure 6 , Figure 6 is a structural block diagram of a water line adaptive generation device 10 provided by an embodiment of the present application, and the water line adaptive generation device 10 comprises:

[0095] A parameter setting module 11 is configured to pre-set parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the water line sending period and the maximum delay period, and initialize the values of the micro-cluster queue and the current delay period.

[0096] A water line generation module 12 is configured to obtain the data amount in the micro-cluster queue, and perform the following logical judgment according to the data amount and the length of the micro-cluster queue.

[0097] If the data amount is less than the length, the event time of the real-time data is obtained and added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as the water line to be sent.

[0098] If the data amount is equal to the length, whether to replace the current event time and send it as the watermark is determined according to the required maximum accuracy, the required minimum accuracy and the maximum delay period;

[0099] If the data amount is greater than the length, the smallest event time in the micro-cluster queue is sent as the watermark;

[0100] The step loop module 13 is configured to return to the step of obtaining the data amount in the micro-cluster queue, and the loop logic judgment process is repeated until the stream processing is completed.

[0101] Optionally, the watermark generation module 12 is further configured to:

[0102] If the event time of the data with the first quantity in the micro-cluster queue is greater than the current watermark, the event time of the data with the first quantity is selected from the micro-cluster queue according to the preset sorting to replace the current event time and send it as the watermark;

[0103] If the event time of the data with the second quantity in the micro-cluster queue is greater than the current watermark and the delay period is greater than the maximum delay period, the event time of the data with the second quantity is selected from the micro-cluster queue according to the preset sorting to replace the current event time and send it as the watermark;

[0104] If the event time of the data with the second quantity in the micro-cluster queue is not greater than the current watermark, the current delay period is accumulated by one, and the current event time is sent as the watermark.

[0105] Optionally, the first quantity is a product of the length of the micro-cluster queue and the required maximum accuracy.

[0106] Optionally, the second quantity is a product of the length of the micro-cluster queue and the required minimum accuracy.

[0107] Optionally, the preset sorting is from small to large.

[0108] Optionally, the required maximum accuracy is set to 100% in advance, which means that all the data in the micro-cluster queue is the data of the next period;

[0109] The required minimum accuracy is set to 98% in advance, which means that the maximum number of data allowed to be lost in a micro-cluster queue is the third quantity.

[0110] The maximum delay period is set to 0 in advance, which means that the watermark is sent regardless of whether there is data in the micro-cluster queue that meets the required minimum accuracy.

[0111] The length of the micro-cluster queue is set according to the data size of each cycle in the measurement data, and is generally not more than the data size of one cycle.

[0112] Optionally, the third quantity is a product of the length of the micro-cluster queue and a difference value; wherein the difference value is a difference between 1 and the required minimum accuracy.

[0113] It is worth noting that the working process of each module in the water level line adaptive generation device 10 described in the embodiments of the present application can refer to the working process of the water level line adaptive generation method described in the above embodiments, which will not be repeated here.

[0114] The water level line adaptive generation device provided by the embodiments of the present application initializes the values of the micro-cluster queue and the current delay cycle by pre-setting the parameters of the length of the micro-cluster queue, the required maximum accuracy, the required minimum accuracy, the water level line sending cycle and the maximum delay cycle; acquires the data size in the micro-cluster queue, and performs the following logical judgment according to the data size and the length of the micro-cluster queue: if the data size is less than the length, the event time of real-time data is added to the micro-cluster queue, and the smallest event time in the micro-cluster queue is taken as the water level line to be sent; if the data size is equal to the length, whether to replace the current event time and take it as the water level line to be sent is judged according to the required maximum accuracy, the required minimum accuracy and the maximum delay cycle; if the data size is greater than the length, the smallest event time in the micro-cluster queue is taken as the water level line to be sent; the step of acquiring the data size in the micro-cluster queue is returned, and the logical judgment process is circularly performed until the stream processing is finished, so that the delay data in the real-time data stream can be processed by flexibly configuring the required maximum accuracy and the required minimum accuracy, the adaptive dynamic generation of the water level line is realized, thereby reducing the loss of delay data, shortening the delay time, and meeting the requirements of real-time and calculation accuracy.

[0115] In addition, the embodiments of the present application also provide a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the above water level line adaptive generation method when running.

[0116] The embodiments of the present application also provide a computer program product, which comprises a computer program; wherein the computer program controls the device where the computer program product is located to execute the above water level line adaptive generation method when running.

[0117] Referring to Figure 7 , Figure 7is a structural block diagram of a computer device 20 provided by an embodiment of the present application, the computer device 20 comprising: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. The processor 21 implements the steps in the water line self-adaptive generation method embodiment when executing the computer program. Alternatively, the processor 21 implements the functions of each module / unit in each device embodiment when executing the computer program.

[0118] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device 20.

[0119] The computer device 20 can include, but is not limited to, the processor 21 and the memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the computer device 20 and does not limit the computer device 20, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the computer device 20 can also include an input / output device, a network access device, a bus, etc.

[0120] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 21 is the control center of the computer device 20, which connects each part of the computer device 20 through various interfaces and lines.

[0121] The memory 22 can be used to store the computer programs and / or modules, and the processor 21 realizes various functions of the computer device 20 by running or executing the computer programs and / or modules stored in the memory 22, and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 22 can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0122] The modules / units integrated in the computer device 20 are stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of the above-mentioned various method embodiments when executed by the processor 21. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0123] The above is the preferred embodiment of the present application, and it should be noted that those skilled in the art can make some improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered to be within the protection scope of the present application.

Claims

1. A water line self-adaptive generation method, characterized in that, The method comprises the following steps: pre-setting parameters of a length of a micro-cluster queue, a required maximum accuracy, a required minimum accuracy, a water line sending period and a maximum delay period, initializing values of the micro-cluster queue and a current delay period; obtaining an amount of data in the micro-cluster queue, and performing the following logical judgment according to the amount of data and the length of the micro-cluster queue: if the amount of data is less than the length, adding an event time of real-time data to the micro-cluster queue, and sending a minimum event time in the micro-cluster queue as a water line; if the amount of data is equal to the length, judging whether to replace a current event time and send it as a water line according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; if the amount of data is greater than the length, sending a minimum event time in the micro-cluster queue as a water line; returning to the step of obtaining the amount of data in the micro-cluster queue, and performing a loop logical judgment process until stream processing is completed; the step of judging whether to replace the current event time and send it as a water line according to the required maximum accuracy, the required minimum accuracy and the maximum delay period comprises the following steps: if there are a first number of data event times greater than the current water line in the micro-cluster queue, selecting the first number of event times in the micro-cluster queue according to a preset sorting to replace the current event time and send it as a water line; the first number is a product of the length of the micro-cluster queue and the required maximum accuracy; if there are a second number of data event times greater than the current water line in the micro-cluster queue and a delay period greater than the maximum delay period, selecting the second number of event times in the micro-cluster queue according to the preset sorting to replace the current event time and send it as a water line; the second number is a product of the length of the micro-cluster queue and the required minimum accuracy; if there is no second number of data event times greater than the current water line in the micro-cluster queue, accumulating the current delay period by one, and sending the current event time as a water line.

2. The water line self-adaptive generation method of claim 1, wherein, The preset sorting is from small to large.

3. The water line self-adaptive generation method of claim 2, wherein, The required maximum accuracy is pre-set as 100%, indicating that all data in the micro-cluster queue are data of the next period; the required minimum accuracy is pre-set as 98%, indicating that a maximum number of third number of data is allowed to be lost in a micro-cluster queue; the maximum delay period is pre-set as 0, indicating that a water line is sent regardless of whether there is data meeting the required minimum accuracy in the micro-cluster queue; the length of the micro-cluster queue is set according to the amount of data of each period in the measurement data, and is generally not more than the amount of data of one period; the third number is a product of the length of the micro-cluster queue and a difference value; the difference value is a difference between 1 and the required minimum accuracy.

4. A water line self-adapting generation apparatus, characterized in that, The method comprises the following steps: a parameter setting module is configured to pre-set parameters of a length of a micro-cluster queue, a required maximum accuracy, a required minimum accuracy, a water line sending period and a maximum delay period, and initialize values of the micro-cluster queue and a current delay period; The water line generation module is configured to obtain an amount of data in the micro-cluster queue, and perform the following logical judgment according to the amount of data and a length of the micro-cluster queue: If the amount of data is less than the length, an event time of real-time data is added to the micro-cluster queue, and a smallest event time in the micro-cluster queue is sent as a water line; If the amount of data is equal to the length, whether to replace a current event time and send it as a water line is determined according to the required maximum accuracy, the required minimum accuracy and the maximum delay period; If the amount of data is greater than the length, the smallest event time in the micro-cluster queue is sent as a water line; The step cycle module is configured to return to the step of obtaining the amount of data in the micro-cluster queue, and perform a loop logical judgment process until stream processing is completed. The determination of whether to replace the current event time and send it as a water line according to the required maximum accuracy, the required minimum accuracy and the maximum delay period includes: If event times of a first number of data in the micro-cluster queue are greater than a current water line, the first number of event times are selected from the micro-cluster queue according to a preset sorting to replace the current event time and send it as a water line; the first number is a product of the length of the micro-cluster queue and the required maximum accuracy; If event times of a second number of data in the micro-cluster queue are greater than the current water line and a delay period is greater than the maximum delay period, the second number of event times are selected from the micro-cluster queue according to the preset sorting to replace the current event time and send it as a water line; the second number is a product of the length of the micro-cluster queue and the required minimum accuracy; If there is no event time of the second number of data in the micro-cluster queue that is greater than the current water line, the current delay period is accumulated by one, and the current event time is sent as a water line.

5. A computer device, comprising: The computer readable storage medium includes a stored computer program; when the computer program is running, the computer readable storage medium controls a device where the computer readable storage medium is located to perform the water line adaptive generation method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored computer program; when the computer program is running, the computer readable storage medium controls a device where the computer readable storage medium is located to perform the water line adaptive generation method according to any one of claims 1-3.

Images