Gaia platform water level dynamic adjustment method and device based on event time window

Abstract

The present application provides a method and device for dynamically adjusting the water level of the Gaia platform based on an event time window, and relates to the field of data processing. The method includes: obtaining the stream data transmitted by the previous operator in the distributed system, wherein the stream data includes the data and the event timestamp when the data is generated; according to the stream data, calculating the water level of the stream data in the operator state; according to the water level cycle, inserting the water level into the stream data for transmission to the next operator, and dynamically adjusting the water level through the change of the stream data in the operator state, thereby ensuring the accuracy and low latency of the system's event processing.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for dynamically adjusting the water level of a Gaia platform based on an event time window. Background Art

[0002] Gaia is an open source computing platform for distributed data stream processing and batch processing, supporting batch data processing and stream distributed data processing in a distributed environment. When processing a continuously arriving and possibly infinite event stream, time and time processing windows become the most core elements of the application. The event time of the data is the actual time when the event in the data stream occurs, and it is based on the timestamp attached to the event in the data stream. Compared with other times, event time completely decouples processing speed and results. Regardless of the processing speed of the data stream and the order in which the event time arrives at the operator, the window based on event time will generate the same result. Among them, the watermark is a global event progress indicator, which indicates a time when the system is sure that there will be no more delayed events, and is used to balance delays and the integrity of results.

[0003] In the prior art, a fixed watermark is preset as a global event progress indicator. When the fixed watermark is an aggressive watermark, low latency of event processing is guaranteed, but the accuracy of event processing is reduced. When the fixed watermark is a conservative watermark, high accuracy of event processing is guaranteed, but processing latency needs to be increased. Currently, when the system faces dynamic and disordered stream data, the fixed watermark cannot be changed dynamically, which is not conducive to high accuracy and low latency of event processing. Summary of the invention

[0004] The purpose of this application is to provide a method, device, equipment and storage medium for dynamically adjusting the water level line of the Gaia platform based on an event time window in response to the above-mentioned deficiencies in the prior art, so as to solve the technical problem that the fixed water level line in the prior art cannot be changed dynamically, which is not conducive to high accuracy and low latency in event processing.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for dynamically adjusting a water level of a Gaia platform based on an event time window, which is applied to a node of a distributed system, and the method includes:

[0007] Obtain the stream data transmitted by the previous operator in the distributed system, where the stream data includes the data and the event timestamp when the data was generated;

[0008] Calculating the water level of the flow data in the operator state according to the flow data;

[0009] According to the watermark period, the watermark is inserted into the stream data for transmission to the next operator.

[0010] Optionally, before calculating the watermark of the stream data in the operator state according to the stream data, the method further includes:

[0011] Assigning operator states to operators, and dividing the stream data in the current operator stream into micro-cluster data based on a data stream time model;

[0012] The event timestamp of the micro-cluster data is stored in the operator state, wherein the operator state stores the event timestamp of the preset data that has most recently flowed through the current operator.

[0013] Optionally, calculating the watermark of the stream data in the operator state according to the stream data includes:

[0014] Calculate and obtain the maximum event timestamp of the micro-cluster data;

[0015] Obtaining a maximum allowable out-of-order time of the micro-cluster data;

[0016] Calculating and obtaining the disorder degree of the micro-cluster data;

[0017] The waterline of the micro-cluster data is calculated based on the maximum event timestamp, the maximum allowed disorder time, the disorder degree, and a preset algorithm.

[0018] Optionally, the calculating and obtaining the maximum event timestamp of the micro-cluster data includes:

[0019] The maximum event timestamp is obtained by comparison according to the event timestamps of the micro-cluster data.

[0020] Optionally, the calculating to obtain the disorder degree of the micro-cluster data includes:

[0021] When new data flows into the operator, the disorder degree at the current moment is calculated according to the timestamp of the micro-cluster data.

[0022] Optionally, the calculating the disorder degree at the current moment according to the timestamp of the micro-cluster data includes:

[0023] Calculating and obtaining the difference between adjacent event timestamps in the micro-cluster data;

[0024] Calculate and obtain the absolute value of the difference between adjacent event timestamps in the micro-cluster data;

[0025] The disorder degree of the micro-cluster data is calculated according to the difference between the timestamps of adjacent events in the micro-cluster data and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data.

[0026] Optionally, when new data flows into the operator, the event timestamp in the micro-cluster data is updated according to the event timestamp of the new data;

[0027] Calculate the local disorder degree at the current time according to the updated micro-cluster data;

[0028] According to the local disorder degree of the current time, the maximum allowable disorder time value of the waterline is adjusted.

[0029] In a second aspect, an embodiment of the present application provides a water level dynamic adjustment device, comprising:

[0030] An acquisition module is used to acquire the stream data transmitted by the previous operator in the distributed system, where the stream data includes the data and the event timestamp when the data is generated;

[0031] A calculation module, used for calculating the water level of the flow data in the operator state according to the flow data;

[0032] The insertion module is used to insert the waterline into the stream data according to the waterline period so as to transmit it to the next operator.

[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method for dynamic adjustment of the water level line of the Gaia platform based on the event time window as described in any one of the first aspects above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is read and executed, the method for dynamically adjusting the water level line of the Gaia platform based on the event time window as described in any one of the first aspects above is implemented.

[0035] Compared with the prior art, this application has the following beneficial effects:

[0036] The embodiments of the present application provide a method, apparatus, device and storage medium for dynamically adjusting the water level of the Gaia platform based on an event time window. The method obtains the stream data transmitted by the previous operator in the distributed system, wherein the stream data includes the data and the event timestamp when the data is generated. The water level of the stream data in the operator state is calculated according to the stream data, and the water level is inserted into the stream data according to the water level period for transmission to the next operator. The method calculates the water level of the stream data in the operator state at the current moment according to the stream data in the operator, and transmits the stream data carrying the water level at the current moment to the next operator. The present application can dynamically adjust the water level according to the change of the stream data in the operator state, thereby ensuring the accuracy and low latency of the system's event processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic diagram of a Gaia platform water level dynamic adjustment system based on an event time window provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a flow chart of a method for dynamically adjusting the water level of a Gaia platform based on an event time window provided in an embodiment of the present application;

[0040] Figure 3 A flowchart of a method for obtaining operator status provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of an operator state provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a flow chart of calculating a water level line of stream data in an operator state provided in an embodiment of the present application;

[0043] Figure 6 A flowchart of a method for calculating the disorder degree of micro-cluster data provided in an embodiment of the present application;

[0044] Figure 7 A flowchart of a method for adjusting the maximum allowable out-of-order time value of a water level provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of various water level triggering window actions provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of a water level dynamic adjustment device provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0050] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0051] Data mining for big data has two most important tasks. The first is real-time performance. For massive event data, it is necessary to analyze it in real time and quickly feedback the results. The second is accuracy. It is necessary to accurately extract the valuable information hidden in the massive event data, and then convert the valuable information into organized knowledge, which is represented by analytical models and other methods. Then, the analytical models are applied to production and life to improve production efficiency, optimize marketing plans, etc. Therefore, for big data computing engines, a high real-time, low-latency, and high-accuracy event data processing strategy is required.

[0052] The embodiment of this application takes the Gaia platform, a big data computing engine, as an example to specifically explain the Gaia platform water level dynamic adjustment method and device based on the event time window proposed in this application. The Gaia platform is a highly efficient and scalable new generation big data analysis system with multiple computing models coexisting. Of course, it can also be other big data computing engines, which are not specifically limited in the embodiment of this application. Figure 1 A schematic diagram of a Gaia platform water level dynamic adjustment system based on an event time window provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the Gaia platform water level dynamic adjustment system includes: events, message queues, and Gaia platform, wherein the Gaia platform is a distributed system that supports distributed processing of streaming data.

[0053] An event includes event production and event time, where the event time is the time when each event is created on its producer, and the event time is also called the event timestamp. Each event produced by a producer carries a corresponding event timestamp. In an embodiment of the present application, there may be multiple producers of events, and the multiple producers may produce multiple events in parallel to obtain stream data of multiple events, which includes the data of the event itself and the event timestamp.

[0054] The stream data may be cached via a message queue, wherein there may be multiple message queues, that is, the stream data may be cached in parallel via multiple message queues.

[0055] The stream data flows into multiple data sources in the Gaia platform through multiple message queues, wherein multiple data sources can be used to cache the stream data to be processed for the Gaia platform, which can be understood as the entrance of the stream data into the Gaia platform. The stream data is distributed to different operators of different nodes in the Gaia platform through the data source, and the allocation logic can be preset. For example, it can be distributed to the corresponding operator of the node that processes the type of stream data according to the type of stream data, and it can also be distributed to the operator of the corresponding node according to the processing method of the stream data. The allocation logic is not specifically limited in this application. Among them, the operator is a mapping from one function space to another function space. In an embodiment of the present application, the operator can process the incoming stream data and other operations, obtain the processed stream data and transmit it to the next operator.

[0056] Since the Gaia platform is a distributed system, there are multiple parallel nodes in the system, and each node contains multiple different operators. Stream data is distributed to different operators of multiple different nodes to achieve parallel processing of stream data. In the process of streaming data flowing from the data source into the data sink window, it will undergo multiple processing and transmissions, that is, it will pass through the operators of multiple nodes. However, due to the different computing resources of each node and the different network transmission speeds, the processing time of the event stream data is different. When entering the operator of the next node, the event timestamps of the stream data are not sorted according to the initial ideal time order. The events corresponding to the stream data will be locally disordered or delayed. Disorder or delay will cause the event time window to be unable to determine whether the stream data has arrived and decide whether the event time window is closed and calculated. Therefore, for the Gaia platform, a local disordered stream data processing strategy with low latency and high accuracy is required.

[0057] The water level is a global event progress indicator, indicating a time when the system is sure that there will be no more delayed events. In the embodiment of the present application, the Gaia platform solves the problem of local disorder or event delay through the water level. When the stream data is allocated to the operator, the water level at the current moment is calculated according to the stream data in the operator, and the stream data carrying the water level at the current moment is transmitted to the next operator. When new stream data flows into the operator, the water level at the current moment is calculated according to the latest stored stream data in the operator. That is, the present application ensures the accuracy and low latency of the system's event processing by dynamically adjusting the water level.

[0058] The system solves the problem of low latency and accuracy when facing data streams with uncertain or elastic disorder of events. At the same time, the system provides reliable guarantees for local data disorder and data delay caused by different computing resources and speeds or different network transmission rates in the distributed Gaia platform.

[0059] The following is a detailed explanation of the implementation process of the Gaia platform water level dynamic adjustment method based on the event time window provided in the embodiment of the present application through multiple examples. Figure 1 It is understood that the multiple examples of this application are specifically explained using the Gaia platform as an example. Of course, the type of big data processing engine is not specifically limited in the application. Figure 2 A schematic diagram of a flow chart of a method for dynamically adjusting the water level of a Gaia platform based on an event time window provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0060] S201, obtaining the stream data transmitted by the previous operator in the distributed system.

[0061] The stream data includes the data and the event timestamp when the data is generated.

[0062] After the user runs the Gaia platform, the streaming data in the data source can be read into the distributed environment according to the pre-configured data source, flow into the operator according to the preset distribution logic, and be processed and transmitted in multiple operators.

[0063] According to the preset relationship and order between operators, when the stream data transmitted by the previous operator flows into the current operator, the stream data stored in the current operator is obtained. The stream data stored in the current operator includes event data and event timestamps.

[0064] S202, calculating the water level of the flow data in the operator state according to the flow data.

[0065] According to the stream data stored in the current operator, that is, the stream data in the operator state, the watermark of the stream data in the current operator state is calculated.

[0066] Among them, the operator state can be represented as the operator's memory, indicating that the operator can store data.

[0067] S203, according to the waterline period, inserting the waterline into the stream data for transmission to the next operator.

[0068] According to the waterline of the stream data in the current operator state calculated in S202 and the waterline period, the waterline is inserted into the stream data in the current operator state, and then transmitted to the next operator to continue processing and transmitting the stream data. The waterline period is a preset fixed value.

[0069] When the event time window receives the transmitted stream data and watermark, the event time window triggers the window to close and calculate according to the watermark value.

[0070] In summary, an embodiment of the present application provides a method for dynamically adjusting the water level of the Gaia platform based on an event time window, by obtaining the stream data transmitted by the previous operator in the distributed system, wherein the stream data includes the data and the event timestamp when the data is generated, and calculating the water level of the stream data in the operator state according to the stream data, and inserting the water level into the stream data according to the water level period for transmission to the next operator. This method calculates the water level of the stream data in the operator state at the current moment according to the stream data in the operator, and transmits the stream data carrying the water level at the current moment to the next operator. It can be understood that when new stream data flows into the operator, the water level of the stream data in the operator state at the current moment is calculated according to the latest stored stream data in the operator, that is, the present application can dynamically adjust the water level by changes in the stream data in the operator state, thereby ensuring the accuracy and low latency of the system's event processing.

[0071] In the above Figure 2 Based on the method for dynamically adjusting the water level of the Gaia platform based on the event time window shown, the embodiment of the present application also provides a method for obtaining the operator state. The embodiment is specifically explained in conjunction with the accompanying drawings as follows. Figure 3 A flowchart of a method for obtaining operator status provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, before the above method S202, before calculating the water level of the flow data in the operator state according to the flow data, it also includes:

[0072] S301, assigning operator states to operators, and dividing stream data in the current operator into micro-cluster data based on a data stream time model.

[0073] Operator states are allocated to operators at different parallelisms, that is, data storage space is allocated to operators, and the stream data in the current operator is divided into micro-cluster data based on the data stream time model.

[0074] In the embodiment of the present application, the micro-cluster data includes a plurality of data, wherein each data represents data of an event and an event timestamp when the event is generated.

[0075] Figure 4 A schematic diagram of an operator state provided in an embodiment of the present application, such as Figure 4 As shown, if there is streaming data in the operator, the operator has been assigned an operator state.

[0076] S302, storing the event timestamp of the micro-cluster data in the operator state.

[0077] The timestamp of the micro-cluster data is stored in the operator state, which is the micro-cluster state within the operator, such as Figure 4 As shown, assuming that the length of the micro-cluster data in the operator is 5, the state of the micro-cluster data in the operator is five event timestamps. Of course, the length of the micro-cluster data can also be other lengths, which can be pre-set according to the density of the event timestamps, and is not specifically limited in the embodiment of the present application.

[0078] The operator state stores the event timestamps of the latest preset data that flows through the current operator in order, such as Figure 4 As shown in the figure, when a new event timestamp flows into the operator, the new event timestamp is stored in the micro-cluster state in the operator, and the old event timestamp is discarded according to time. If the event timestamp with the number 7 flows into the current operator, the event timestamp with the number 2 is discarded, that is, the micro-cluster state in the current operator is 7, 6, 5, 4, 3.

[0079] The embodiment of the present application allocates an operator state to the operator and stores the event timestamp of the micro-cluster data in the operator state to store the event timestamp of the latest preset sub-data flowing through the current operator, and then calculates the watermark of the flow data in the operator state, so that the real-time performance of the watermark is higher and the delay of event processing is reduced.

[0080] In the above Figure 2 Based on the method for dynamically adjusting the water level of the Gaia platform based on the event time window shown, the embodiment of the present application also provides a method for implementing the water level of the stream data in the calculation operator state. The embodiment is specifically explained in conjunction with the accompanying drawings as follows. Figure 5 A schematic diagram of a flow chart of calculating the water level of stream data in the operator state provided in an embodiment of the present application. Figure 5 As shown, the above method S202, based on the stream data, calculates the water level of the stream data in the operator state, including:

[0081] S501, calculating and obtaining the maximum event timestamp of micro-cluster data.

[0082] like Figure 3 In the illustrated embodiment, the event timestamps of the micro-cluster data are stored in the operator state, and the largest event timestamp is selected from the event timestamps of the micro-cluster data to calculate the watermark of the micro-cluster data in the operator state.

[0083] S502: Obtain the maximum allowable out-of-order time of micro-cluster data.

[0084] The maximum allowed out-of-order time of micro-cluster data is determined according to the event time density.

[0085] Because some data in the stream data arrives late, that is, the event timestamps of the micro-cluster data do not arrive in order, the event timestamps of the micro-cluster data are out of order. Therefore, the maximum allowed out-of-order time is set to allow data to arrive delayed for a certain period of time. If the event time is dense, that is, the event timestamps are close, then the maximum allowed out-of-order time can be set smaller. If the event time is scattered, that is, the event timestamps have a large gap, then the maximum allowed out-of-order time can be set larger.

[0086] S503, calculating and obtaining the disorder degree of the micro-cluster data.

[0087] The disorder degree of the micro-cluster data is calculated according to the event timestamp of the micro-cluster data in the operator state. The disorder degree ranges from 0 to 1. If the disorder degree is 0, it means that the micro-cluster data is completely in reverse order, and if the disorder degree is 1, it means that the micro-cluster data is completely in normal order.

[0088] S504: Calculate and obtain the watermark of the micro-cluster data according to the maximum event timestamp, the maximum allowed disorder time, the disorder degree, and a preset algorithm.

[0089] The waterline of the micro-cluster data is calculated based on the maximum event timestamp obtained in S501, the maximum allowed disorder time obtained in S502, the disorder degree obtained in S503, and the preset algorithm. The preset algorithm can be shown as formula (1):

[0090] W=NM*p Formula (1)

[0091] Among them, W represents the watermark value, N represents the current maximum event timestamp, M represents the maximum allowed out-of-order time, and p represents the degree of out-of-order.

[0092] In the embodiment of the present application, the watermark value is obtained by subtracting the product of the maximum allowed disorder time and the disorder degree from the current maximum event timestamp, so that the watermark is dynamically adjusted according to the event time disorder program of the stream data, thereby ensuring high accuracy and low latency of the system.

[0093] In the above Figure 5Based on the water level line of the stream data in the calculation operator state shown, the embodiment of the present application also provides an implementation method for calculating the maximum event timestamp of the micro-cluster data. In the above S501, the maximum event timestamp of the micro-cluster data is calculated, including: obtaining the maximum event timestamp by comparison according to the event timestamp of the micro-cluster data.

[0094] like Figure 4 As shown in the figure, if the micro-cluster states in the current operator are 6, 5, 4, 3, and 2, the maximum event timestamp obtained by comparison is 6.

[0095] In the embodiment of the present application, the event timestamps of the micro-cluster data can be sorted by a sorting algorithm to obtain the maximum event timestamp. The sorting algorithm can be bubble sort, selection sort, insertion sort, shell sort, merge sort, quick sort, heap sort, etc., which are not specifically limited in the embodiment of the present application.

[0096] Of course, other algorithms may also be used to compare the event timestamps of the micro-cluster data to obtain the maximum event timestamp, and the comparison method is not specifically limited in the embodiment of the present application.

[0097] In the above Figure 5 Based on the water level line of the flow data in the calculation operator state shown, the embodiment of the present application also provides a method for calculating the disorder degree of micro-cluster data. In the above S503, the disorder degree of the micro-cluster data is calculated, including: when new data flows into the operator, the disorder degree at the current moment is calculated according to the event timestamp of the micro-cluster data.

[0098] like Figure 4 As shown, if data with event timestamp 7 flows into the operator, and the event timestamps of the micro-cluster data in the current operator are 7, 6, 5, 4, and 3, the disorder degree of the micro-cluster data in the operator state at the current moment is calculated according to the event timestamps of the micro-cluster data in the current operator.

[0099] On the basis of the above-mentioned embodiment, the embodiment of the present application further provides another method for calculating and obtaining the disorder degree of micro-cluster data. The embodiment is specifically explained below in conjunction with the accompanying drawings. Figure 6 A flow chart of a method for calculating the disorder degree of micro-cluster data provided in an embodiment of the present application. Figure 6 As shown, in the above embodiment, the disorder degree at the current moment is calculated according to the event timestamp of the micro-cluster data, including:

[0100] S601, calculating and obtaining the difference between adjacent event timestamps in micro-cluster data.

[0101] As described in the above embodiment, if the event timestamps of the micro-cluster data in the current operator are 7, 6, 5, 4, 3, where the order in which the stream data flows into the operator is 3, 4, 5, 6, 7, then when calculating the difference between adjacent event timestamps in the micro-cluster data, the timestamp of the adjacent event is subtracted from the timestamp of the event that flows into the operator the latest, that is, the differences between the adjacent event timestamps corresponding to the event timestamps 7, 6, 5, 4, 3 of the micro-cluster data in the current operator are 1(7-6), 1(6-5), 1(5-4), 1(4-3), respectively.

[0102] S602: Calculate and obtain the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data.

[0103] As described in S601, the differences between adjacent event timestamps corresponding to event timestamps 7, 6, 5, 4, and 3 in the micro-cluster data in the current operator are 1(7-6), 1(6-5), 1(5-4), and 1(4-3), respectively. Then, the absolute values ​​of the differences between adjacent event timestamps in the micro-cluster data in the current operator are 1, 1, 1, and 1, respectively.

[0104] S603: Calculate the disorder degree of the micro-cluster data according to the difference between the timestamps of adjacent events in the micro-cluster data and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data.

[0105] The difference between the timestamps of adjacent events in the micro-cluster data is summed, and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data is also summed. The results of the two summations are divided to calculate the disorder of the micro-cluster data, as shown in formula (2):

[0106]

[0107] Among them, T i represents the i-th event timestamp in the micro-cluster data, and p represents the disorder degree of the current micro-cluster data, i.e., the disorder degree.

[0108] For example, if the event timestamps of the micro-cluster data in the current operator are 7, 6, 5, 4, and 3, then T5=7, T4=6, T3=5, T2=4, and T1=3, then p=1 is calculated by formula (2), indicating that the disorder degree of the micro-cluster data of the operator at the current moment is 1, that is, completely positive order.

[0109] The embodiment of the present application calculates the disorder degree of the current moment through the difference between the timestamps of adjacent events in the micro-cluster data. By continuously obtaining the disorder degree of the current moment, the water level of the current micro-cluster data can be continuously calculated based on the continuously obtained disorder degree of the current moment and a preset algorithm, thereby realizing dynamic adjustment of the water level and ensuring high accuracy and low latency of the system.

[0110] On the basis of the above-mentioned embodiment, the embodiment of the present application further provides a method for adjusting the maximum allowable disorder time value of the water level line, and the embodiment is specifically explained in conjunction with the accompanying drawings as follows. Figure 7 A flow chart of a method for adjusting the maximum allowable out-of-order time value of a water level provided in an embodiment of the present application. Figure 7 As shown, in the above embodiment, it also includes:

[0111] S701: When new data flows into the operator, the event timestamp of the micro-cluster data is updated according to the event timestamp of the new data.

[0112] exist Figure 6 Based on the embodiment, the event timestamps of the micro-cluster data in the operator are 7, 6, 5, 4, and 3. When new data flows into the operator, if the event timestamp of the new data is 8, the event timestamp of the micro-cluster data is updated according to the event timestamp of the new data, and the event timestamp of the updated micro-cluster data is 8, 7, 6, 5, and 4.

[0113] S702: Calculate the local disorder degree at the current time according to the updated micro-cluster data.

[0114] According to the event timestamps 8, 7, 6, 5, and 4 of the updated micro-cluster data, the local disorder degree at the current time is calculated by formula (2).

[0115] S703, adjusting the maximum allowable disorder time value of the waterline according to the local disorder degree of the current time.

[0116] The maximum allowable out-of-order time value of the water level, i.e. the water level value, is adjusted by formula (1).

[0117] The embodiment of the present application adjusts the maximum allowable disorder time value of the water level according to the local disorder degree of the current time, so that the maximum allowable waiting time changes dynamically, thereby ensuring the calculation accuracy of the system while reducing the calculation delay.

[0118] Figure 8 A schematic diagram of various water level triggering window actions provided in the embodiments of the present application, such as Figure 8 As shown, when the water level is set too large, such as Window1, the event time window will be invalidly waiting, thereby reducing the calculation speed of the system; when the water level is set too small, such as Window2, the event time window will lose data, affecting the accuracy of the event time window calculation. The present application provides a Gaia platform water level dynamic adjustment method based on event time window, such as Window3, which can make the water level change dynamically, ensuring the calculation accuracy of the system while reducing the calculation delay.

[0119] The following is a description of a water level line dynamic adjustment device provided by the present application for execution. Its specific implementation process and technical effects are mentioned above and will not be repeated below.

[0120] Fig. 9 A schematic diagram of a water level dynamic adjustment device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the water level dynamic adjustment device includes:

[0121] The acquisition module 901 is used to acquire the stream data transmitted by the previous operator in the distributed system, wherein the stream data includes the data and the event timestamp when the data is generated.

[0122] The calculation module 902 is used to calculate the water level of the flow data in the operator state according to the flow data.

[0123] The insertion module 903 is used to insert the waterline into the stream data according to the waterline period so as to transmit it to the next operator.

[0124] Optionally, the computing module 902 is further used to assign an operator state to the operator, and divide the stream data in the current operator into micro-cluster data based on the data flow time model; store the event timestamp of the micro-cluster data in the operator state, wherein the operator state stores the event timestamp of the latest preset data flowing through the current operator.

[0125] Optionally, the calculation module 902 is used to calculate the maximum event timestamp of the micro-cluster data; obtain the maximum allowable disorder time of the micro-cluster data; calculate the disorder degree of the micro-cluster data; and calculate the waterline of the micro-cluster data according to the maximum event timestamp, the maximum allowable disorder time, the disorder degree and a preset algorithm.

[0126] Optionally, the calculation module 902 is configured to obtain a maximum event timestamp by comparison according to the event timestamps of the micro-cluster data.

[0127] Optionally, the calculation module 902 is used to calculate the disorder degree at the current moment according to the event timestamp of the micro-cluster data when new data flows into the operator.

[0128] Optionally, the calculation module 902 is used to calculate the difference between adjacent event timestamps in the micro-cluster data; calculate the absolute value of the difference between adjacent event timestamps in the micro-cluster data; and calculate the disorder degree of the micro-cluster data based on the difference between adjacent event timestamps in the micro-cluster data and the absolute value of the difference between adjacent event timestamps in the micro-cluster data.

[0129] Optionally, the calculation module 902 is further used to update the event timestamp of the micro-cluster data according to the event timestamp of the new data when new data flows into the operator; calculate the local disorder degree of the current time according to the updated micro-cluster data; and adjust the maximum allowable disorder time value of the watermark according to the local disorder degree of the current time.

[0130] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0131] Fig.10 A schematic diagram of a computer device provided in an embodiment of the present application, wherein the computer device may be a computing device or a server with computing processing capabilities.

[0132] The computer device includes: a processor 101 , a storage medium 102 , and a bus 103 . The processor 101 and the storage medium 102 are connected via the bus 103 .

[0133] The storage medium 102 is used to store programs, and the processor 101 calls the programs stored in the storage medium 102 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0134] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute the above method embodiment when executed by a processor.

[0135] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for dynamically adjusting the water level of a Gaia platform based on an event time window, characterized in that: Applied to a node of a distributed system, the method comprises: Obtain the stream data transmitted by the previous operator in the distributed system, where the stream data includes the data and the event timestamp when the data was generated; Calculating the water level of the flow data in the operator state according to the flow data; inserting the waterline into the stream data according to the waterline period for transmission to the next operator; Before calculating the water level of the stream data in the operator state according to the stream data, the method further includes: Assigning operator states to operators, and dividing the stream data in the current operator into micro-cluster data based on a data stream time model; storing the event timestamp of the micro-cluster data in the operator state, wherein the operator state stores the event timestamp of the preset data that has most recently flowed through the current operator; The step of calculating the watermark of the stream data in the operator state according to the stream data includes: Calculate and obtain the maximum event timestamp of the micro-cluster data; Obtaining a maximum allowable out-of-order time of the micro-cluster data; Calculating and obtaining the disorder degree of the micro-cluster data; Calculate and obtain the watermark of the micro-cluster data according to the maximum event timestamp, the maximum allowed disorder time, the disorder degree, and a preset algorithm; The calculating and obtaining the disorder degree of the micro-cluster data includes: When new data flows into the operator, the disorder degree at the current moment is calculated according to the event timestamp of the micro-cluster data; The calculating the disorder degree at the current moment according to the event timestamp of the micro-cluster data includes: Calculating and obtaining the difference between adjacent event timestamps in the micro-cluster data; Calculate and obtain the absolute value of the difference between adjacent event timestamps in the micro-cluster data; Calculating the disorder degree of the micro-cluster data according to the difference between the timestamps of adjacent events in the micro-cluster data and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data; The method further comprises: When new data flows into the operator, the event timestamp of the micro-cluster data is updated according to the event timestamp of the new data; Calculate the local disorder degree at the current time according to the updated micro-cluster data; According to the local disorder degree of the current time, adjust the maximum allowable disorder time value of the waterline; The preset algorithm includes: W=NM*p; wherein W represents the waterline value, N represents the current maximum event timestamp, M represents the maximum allowed disorder time, and p represents the disorder degree; The calculating the disorder degree of the micro-cluster data according to the difference between the timestamps of adjacent events in the micro-cluster data and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data comprises: according to the formula The disorder degree of the micro-cluster data is calculated; wherein, T i represents the i-th event timestamp in the micro-cluster data, p represents the disorder degree of the current micro-cluster data, and k represents the total number of event timestamps in the micro-cluster data.

2. The method according to claim 1, characterized in that The calculating and obtaining the maximum event timestamp of the micro-cluster data includes: The maximum event timestamp is obtained by comparison according to the event timestamps of the micro-cluster data.

3. A water level dynamic adjustment device, characterized in that: include: An acquisition module is used to acquire the stream data transmitted by the previous operator in the distributed system, where the stream data includes the data and the event timestamp when the data is generated; A calculation module, used for calculating the water level of the flow data in the operator state according to the flow data; An insertion module, used for inserting the waterline into the stream data according to the waterline period to transmit to the next operator; The calculation module is further used to allocate operator states to operators, and divide the flow data in the current operator into micro-cluster data based on the data flow time model; store the event timestamp of the micro-cluster data in the operator state, wherein the operator state stores the event timestamp of the latest preset data flowing through the current operator; calculate the maximum event timestamp of the micro-cluster data; obtain the maximum allowable disorder time of the micro-cluster data; calculate the disorder degree of the micro-cluster data; calculate the water level of the micro-cluster data according to the maximum event timestamp, the maximum allowable disorder time, the disorder degree and the preset algorithm; when new data flows into the operator, calculate the water level of the micro-cluster data according to the micro-cluster data. The event timestamp of the cluster data is used to calculate the disorder degree of the current moment; the difference between adjacent event timestamps in the micro-cluster data is calculated; the absolute value of the difference between adjacent event timestamps in the micro-cluster data is calculated; the disorder degree of the micro-cluster data is calculated based on the difference between adjacent event timestamps in the micro-cluster data and the absolute value of the difference between adjacent event timestamps in the micro-cluster data; when new data flows into the operator, the event timestamp of the micro-cluster data is updated according to the event timestamp of the new data; the local disorder degree of the current time is calculated according to the updated micro-cluster data; according to the local disorder degree of the current time, the maximum allowable disorder time value of the watermark is adjusted; The preset algorithm includes: W=NM*p; wherein W represents the waterline value, N represents the current maximum event timestamp, M represents the maximum allowed disorder time, and p represents the disorder degree; The calculating the disorder degree of the micro-cluster data according to the difference between the timestamps of adjacent events in the micro-cluster data and the absolute value of the difference between the timestamps of adjacent events in the micro-cluster data comprises: according to the formula The disorder degree of the micro-cluster data is calculated; wherein, T i represents the i-th event timestamp in the micro-cluster data, p represents the disorder degree of the current micro-cluster data, and k represents the total number of event timestamps in the micro-cluster data.

4. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method for dynamically adjusting the water level line of the Gaia platform based on the event time window as described in any one of claims 1 to 2 above is implemented.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed, the method for dynamically adjusting the water level line of the Gaia platform based on the event time window described in any one of claims 1 to 2 is implemented.

Images