Method, device and readable storage medium for processing out-of-order data

Abstract

The present application relates to the field of distributed computing and provides a method, device, and computer-readable storage medium for processing out-of-order data, so that the set delay time responds objectively, reasonably, and promptly to the fluctuations of data when processing out-of-order data. The method comprises: calculating the absolute difference between the timestamps of each of n out-of-order data and the previous non-out-of-order data in the last event processing cycle to obtain n-1 timestamp absolute differences; determining the delay time T according to the n-1 timestamp absolute differences; d ; In the current event processing cycle, according to the timestamp and delay time T of the data received in the current time window d , determine the current latest water level; according to the current latest water level, the right boundary of the current time window and the delay time T d , determine whether to trigger the current time window to start processing the data received in the current time window. The technical solution of this application makes the calculation of the latest watermark more objective and also makes the response to data disorder more timely.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing, and in particular to a method, device and readable storage medium for processing out-of-order data. Background Art

[0002] As a distributed computing framework, Flink plays a vital role in real-time big data processing. For processing out-of-order data, Flink offers a range of approaches, primarily through watermarks, permissible delays, and side-outputs. Among these approaches, watermark setting is particularly crucial. A reasonable permissible delay ensures a more rational and rigorous process. However, existing technologies rely on R&D personnel's experience when setting watermarks. This approach, however, is subjective and can be time-consuming, resulting in unscientific and irrational settings. Summary of the Invention

[0003] The present application provides a method, device, and computer-readable storage medium for processing out-of-order data, so that the set delay time responds objectively, reasonably, and promptly to fluctuations in data when processing out-of-order data.

[0004] In one aspect, the present application provides a method for processing out-of-order data, the method comprising:

[0005] Calculate the absolute difference between the timestamps of each piece of n out-of-order data and the previous piece of non-out-of-order data in the previous event processing cycle, and obtain n-1 timestamp absolute differences, where n is a natural number not less than 20;

[0006] Determine the delay time T based on the absolute difference of the n-1 timestamps d ;

[0007] In the current event processing cycle, according to the timestamp of the data received in the current time window and the delay time T d , determine the current latest water level;

[0008] According to the current latest waterline, the right boundary of the current time window and the delay time T d , determine whether to trigger the current time window to start processing the data received in the current time window.

[0009] Optionally, the delay time T is determined based on the absolute difference of the n-1 timestamps. d , including: sorting the n-1 timestamp absolute differences in order of size; determining the timestamp absolute difference corresponding to the specified sequence number as the delay time T d, the specified sequence number is less than the maximum sequence number in the sort.

[0010] Optionally, the delay time T is determined based on the absolute difference of the n-1 timestamps. d , including: sorting the n-1 timestamp absolute differences in order of size; determining the timestamp absolute difference corresponding to the largest sort number in the sort as the delay time T d .

[0011] Optionally, the delay time T is determined based on the absolute difference of the n-1 timestamps. d , including: obtaining the normal distribution N(μ, σ) of the absolute differences of the n-1 timestamps 2 ); the normal distribution N(μ, σ 2 ) corresponds to the delay time T d .

[0012] Optionally, the timestamp of the data received in the current time window and the delay time T d , determine the current latest waterline, including: comparing the timestamp of each data received in the current time window with the delay time T d Calculate the difference to obtain the difference value corresponding to each data received in the current time window; and determine the largest difference value among the differences as the current latest waterline.

[0013] Optionally, the current latest waterline, the right boundary of the current time window and the delay time T d , determine whether to trigger the current time window to start processing the currently received data, including: obtaining the data D received in the current time window current The current latest water level and the delay time T d If the difference is equal to the right boundary of the current time window, the current time window is triggered to start processing the data received in the current time window.

[0014] Optionally, the method further includes: if in the data D current After receiving other data, the data D is abandoned. current Processing of other data received thereafter; or if the difference is greater than the right boundary of the current time window, abandoning the data D current processing.

[0015] On the other hand, the present application provides a device for processing out-of-order data, the device comprising:

[0016] A calculation module is used to calculate the absolute difference between the timestamps of each piece of n out-of-order data and the previous piece of non-out-of-order data in the previous event processing cycle, and obtain n-1 absolute differences in the timestamps, where n is a natural number not less than 20;

[0017] The first determining module is used to determine the delay time T according to the absolute difference of the n-1 timestamps. d ;

[0018] The second determination module is used to determine, in the current event processing cycle, the timestamp of the data received in the current time window and the delay time T d , determine the current latest water level;

[0019] The third determining module is used to determine the current latest waterline, the right boundary of the current time window and the delay time T d , determine whether to trigger the current time window to start processing the data received in the current time window.

[0020] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of the method for processing out-of-order data as described in any of the above embodiments by calling the computer program stored in the memory.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for loading by a processor to execute the steps in the method for processing out-of-order data as described in any of the above embodiments.

[0022] From the technical solution provided by the present application, it can be seen that, on the one hand, due to the delay time T d It is calculated based on the absolute difference of n-1 timestamps. The absolute difference of n-1 timestamps is calculated strictly according to the timestamp between each piece of n out-of-order data and the last piece of non-out-of-order data in the previous event processing cycle, that is, the delay time T d The acquisition of is objective, which provides an objective basis for the subsequent calculation of the latest water level, avoiding the existing technology relying on the experience of R&D personnel when determining the water level. Therefore, according to the current latest water level, the right boundary of the current time window and the delay time T d It is reasonable to determine whether to trigger the current time window to start processing the data received in the current time window; on the other hand, the latest watermark and whether to trigger the current time window to start processing the data received in the current time window are both based on the event processing cycle. Therefore, the response to data fluctuations, that is, data disorder, is relatively timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 is a flowchart of a method for processing out-of-order data provided by an embodiment of the present application;

[0025] Figure 2 Schematic diagram of the structure of the device for processing out-of-order data provided by an embodiment of the present application;

[0026] Figure 3 It is a structural diagram of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] In this specification, adjectives such as first and second may be used only to distinguish one element or action from another element or action, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0029] In this specification, for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportions.

[0030] This application embodiment proposes a method for processing out-of-order data, which can be applied to each computing node of the Flink framework. Figure 1 As shown, it mainly includes steps S101 to S104, which are described in detail as follows:

[0031] Step S101: Calculate the absolute difference in timestamps between each piece of n out-of-order data and the last piece of non-out-of-order data in the last event processing cycle to obtain n-1 absolute differences in timestamps, where n is a natural number not less than 20.

[0032] In an embodiment of the present application, the event processing cycle can be 30 minutes, one hour or one day, etc., and can be set according to business needs. For example, if the data disorder at the current stage is within an acceptable range, the event processing cycle can be appropriately increased, otherwise, the event processing cycle can be reduced. The disordered data in the embodiment of the present application refers to the fact that in the Flink computing framework, due to network and other reasons, the data generated first arrives at the Flink computing framework later than the data generated later and is processed. In other words, the event time (Event Time) corresponding to data A is earlier than the event time corresponding to data B, however, the processing time (Process Time) corresponding to data A is later than the processing time corresponding to data B; the data in the disordered data can be high-density business data, such as log data, and in the log data, it can further be login log data and logout log data.

[0033] Step S102: Determine the delay time T based on the absolute difference of n-1 timestamps d .

[0034] In the embodiment of the present application, the delay time T d It is a parameter associated with the watermark in the Flink computing framework for processing out-of-order data. As mentioned above, due to various reasons, the data corresponding to the event may become out-of-order. That is, in real-time streaming scenarios, data out-of-order is inevitable. Therefore, in order to avoid affecting the system as a whole, it is not appropriate to abandon delayed data in a one-size-fits-all manner. A mechanism is needed to ensure that some delayed data, especially data with a delay time within a certain range, still has a chance to be processed. This mechanism is to set a delay time T associated with the watermark. d The delay time T d , which can be stored in storage media such as Redis and MySQL, and can be broadcast to each computing node of the Flink computing framework through the Broadcast State function or service provided by the Flink framework.

[0035] As an embodiment of the present application, the delay time T is determined based on the absolute difference of n-1 timestamps. d It can be: sort the absolute differences of n-1 timestamps in order of size; determine the absolute difference of the timestamp corresponding to the specified sequence number as the delay time T d , where the specified sequence number is less than the maximum sequence number in the sort. For the absolute difference of n-1 timestamps sorted in order of size, if the absolute difference of the timestamp corresponding to the specified sequence number is determined as the delay time T d , which means that the data corresponding to the event, its maximum delay in reaching the Flink computing framework cannot exceed the delay time T dOtherwise, it will be abandoned. Therefore, if the number of timestamp absolute differences corresponding to the specified sequence number and its previous sequence number is divided by the total number of timestamp absolute differences participating in the sorting, a percentage is obtained. This percentage is also the maximum proportion of disordered data that the system can tolerate within an event processing cycle to all data processed within the event processing cycle. For example, assuming that there are 100 timestamp absolute differences participating in the sorting in order of size, if the timestamp absolute difference corresponding to sequence number 95 is determined as the delay time T d , then the number of timestamp absolute differences corresponding to sequence number 95 and its previous sequence numbers (95) is divided by the total number of timestamp absolute differences participating in the sorting (100), and the resulting proportion is 95%, which means that the maximum proportion of out-of-order data that the system can tolerate in an event processing cycle is 95% of all data processed in the event processing cycle.

[0036] As another embodiment of the present application, the delay time T is determined based on the absolute difference of n-1 timestamps. d It can be: sort the n-1 timestamp absolute differences in order of size; determine the timestamp absolute difference corresponding to the largest sort number in the sort as the delay time T d Different from the above embodiment, this embodiment directly determines the absolute difference of the timestamp corresponding to the largest sequence number in the sequence as the delay time T. d This means that the system has increased tolerance for the delay in the data corresponding to the event reaching the Flink computing framework.

[0037] As another embodiment of the present application, the delay time T is determined based on the absolute difference of n-1 timestamps. d It can be: find the normal distribution N(μ, σ) of the absolute difference of n-1 timestamps 2 ); the normal distribution N(μ, σ 2 ) is determined as the absolute difference of the timestamp corresponding to the mean μ as the delay time T d According to the definition of normal distribution, if the absolute differences of n-1 timestamps obey the normal distribution, it means that the absolute difference of timestamps corresponding to the mean μ has the maximum confidence. Therefore, the normal distribution N(μ,σ 2 ) is determined as the absolute difference of the timestamp corresponding to the mean μ as the delay time T d It has greater rationality.

[0038] Step S103: In the current event processing cycle, according to the timestamp and delay time T of the data received in the current time window d , determine the current latest water level.

[0039] In the embodiment of the present application, the data received in the current time window may contain out-of-order data. For example, the data received in the current time window includes four data, A, B, C, and D, where the event time of data A is ET a The time it takes to reach the Flink computing framework is IT a ; The event time of data B is ET b The time it takes to reach the Flink computing framework is IT b ; The event time of data C is ET c The time it takes to reach the Flink computing framework is IT c ; The event time of data D is ET d The time it takes to reach the Flink computing framework is IT d If ET a <ET b <ET c <ET d , but IT a <IT b <IT d <IT c , it means that the data D is out of order. In addition, it should be noted that the timestamp of the data is also the time when the event corresponding to the data occurs, that is, the event time. For example, the event time ET of the four data A, B, C and D in the above example is a ET b ET c and ET d These are also the timestamps of data A, B, C, and D respectively.

[0040] When the current event processing cycle arrives, according to the delay time T of the Broadcast State function or service broadcast d , and combined with the timestamp of the data received in the current time window, the current latest waterline can be determined. As an embodiment of the present application, according to the timestamp of the data received in the current time window and the delay time T d , the current latest watermark can be determined by comparing the timestamp of each data received in the current time window with the delay time T d Calculate the difference and get the difference corresponding to each data received in the current time window; determine the largest difference among the differences as the current latest watermark. For example, in the example above, data A, B, C, and D arrive at the Flink computing framework in the order of data A arriving first, data B arriving second, data D arriving third, and data C arriving last. When data A arrives, ET a Subtract T d , we get a difference Δt1=ET a -T d, Δt1 is the current water level, recorded as WM1=ETa-T d ; When data B arrives, ET b Subtract T d , we get a difference Δt2=ET b -T d , Δt2 is the current water level, recorded as WM2=ET b -T d ; When data D arrives, ET d Subtract T d , we get a difference Δt4=ET d -T d , Δt4 is the current water level, recorded as WM4=ET d -T d ; When data C arrives, ETc minus T d , we get a difference Δt3=ET c -T d , Δt3 is the current water level, recorded as WM3=ET c -T d Find the maximum value among WM1, WM2, WM3, and WM4, that is, max{WM1, WM2, WM3, WM4}. If WM3 = max{WM1, WM2, WM3, WM4}, then determine WM3 as the current latest waterline.

[0041] Step S104: Based on the current latest water level, the right boundary of the current time window and the delay time T d , determines whether to trigger the current time window to start processing the data received in the current time window.

[0042] In an embodiment of the present application, the time window can be a sliding window or a tumbling window. Whether it is a sliding window or a tumbling window, its essence is an interval that is closed on the left and open on the right. The left endpoint of the interval can be called the left boundary of the window, and the right endpoint can be called the right boundary of the window. As an embodiment of the present application, according to the current latest waterline, the right boundary of the current time window and the delay time T d , to determine whether to trigger the current time window to start processing the data received in the current time window can be: obtain the data D received in the current time window current The current latest water level and delay time T d If the difference is equal to the right boundary of the current time window, the current time window is triggered to start processing the data received in the current time window. For example, if the left boundary and right boundary of the current time window are respectively denoted as W st and W et, receive data D in the current time window current The current latest water level is recorded as WM new , then when W et =WM new -T d When the current time window is triggered, the data received in the current time window is processed. It should be noted that the triggering of the current time window here to start processing the data received in the current time window includes not only the data received in the current time window D current , also included in the data D current For example, for the above embodiment, the event time is ET a ET b ET c and ET d Data A, B, C and D, because the data A, B and D received before data C are received do not meet the current waterline and delay time T d The difference between them is not equal to the right boundary of the current time window. If the current time window is not triggered when receipts A, B and D are received, the current time window will not be triggered. When data C is received, the current latest water level WM3 is obtained because it is different from the delay time T. d The difference between them is exactly equal to the right boundary of the current time window, that is, WM3-T d =W et , then the current time window is triggered and the received data A, B, D and C begin to be processed.

[0043] In the above embodiment, if the data D is received current If other data is received later, the current Processing of other data received later, or if data D current The current latest water level and delay time T d If the difference between them is greater than the right boundary of the current time window, the data D is abandoned. current For example, for the above embodiment, the event times are ET a ET b ET c and ET d If data A, B, C and D are received after data C, the current time window has been triggered to start processing the data received in the current time window and the current time window is closed when data C is received, so the processing of data E is abandoned; or, if WM3-T d >W et , the processing of data C will be abandoned.

[0044] From the above attached Figure 1The example method of processing disordered data shows that, on the one hand, due to the delay time T d It is calculated based on the absolute difference of n-1 timestamps. The absolute difference of n-1 timestamps is calculated strictly according to the timestamp between each piece of n out-of-order data and the last piece of non-out-of-order data in the previous event processing cycle, that is, the delay time T d The acquisition of is objective, which provides an objective basis for the subsequent calculation of the latest water level, avoiding the existing technology relying on the experience of R&D personnel when determining the water level. Therefore, according to the current latest water level, the right boundary of the current time window and the delay time T d It is reasonable to determine whether to trigger the current time window to start processing the data received in the current time window; on the other hand, the latest watermark and whether to trigger the current time window to start processing the data received in the current time window are both based on the event processing cycle. Therefore, the response to data fluctuations, that is, data disorder, is relatively timely.

[0045] Please see the attached Figure 2 , is a device for processing out-of-order data provided in an embodiment of the present application. The device can be applied to a node device in a Flink computing framework. The device includes a computing module 201, a first determining module 202, a second determining module 203, and a third determining module 204, which are described in detail as follows:

[0046] The calculation module 201 is configured to calculate the absolute difference between the timestamps of each piece of n out-of-order data and the previous piece of non-out-of-order data in the previous event processing cycle, and obtain n-1 absolute differences in the timestamps, where n is a natural number not less than 20.

[0047] The first determining module 202 is configured to determine the delay time T according to the absolute difference of n-1 timestamps. d ;

[0048] The second determining module 203 is used to determine, in the current event processing cycle, the timestamp and delay time T of the data received in the current time window. d , determine the current latest water level;

[0049] The third determining module 204 is used to determine the current latest water level, the right boundary of the current time window and the delay time T d , determines whether to trigger the current time window to start processing the data received in the current time window.

[0050] From the description of the above technical solution, it can be seen that, on the one hand, due to the delay time T dIt is calculated based on the absolute difference of n-1 timestamps. The absolute difference of n-1 timestamps is calculated strictly according to the timestamp between each piece of n out-of-order data and the last piece of non-out-of-order data in the previous event processing cycle, that is, the delay time T d The acquisition of is objective, which provides an objective basis for the subsequent calculation of the latest water level, avoiding the existing technology relying on the experience of R&D personnel when determining the water level. Therefore, according to the current latest water level, the right boundary of the current time window and the delay time T d It is reasonable to determine whether to trigger the current time window to start processing the data received in the current time window; on the other hand, the latest watermark and whether to trigger the current time window to start processing the data received in the current time window are both based on the event processing cycle. Therefore, the response to data fluctuations, that is, data disorder, is relatively timely.

[0051] Optionally, the above-mentioned Figure 2 The exemplary first determining module 202 may include a sorting unit and a first delay time determining unit, wherein:

[0052] A sorting unit, configured to sort the n-1 timestamp absolute differences in order of size;

[0053] The first delay time determination unit is used to determine the absolute difference of the timestamps corresponding to the specified sequence number as the delay time T d , where the specified sequence number is smaller than the maximum sequence number in the sort.

[0054] Optionally, the above-mentioned Figure 2 The exemplary first determining module 202 may include a sorting unit and a second delay time determining unit, wherein:

[0055] A sorting unit, configured to sort the n-1 timestamp absolute differences in order of size;

[0056] The second delay time determination unit is used to determine the absolute difference of the timestamp corresponding to the largest sort number in the sort as the delay time T d .

[0057] Optionally, the above-mentioned Figure 2 The exemplary first determination module 202 may include a determining unit and a third delay time determining unit, wherein:

[0058] The calculation unit is used to calculate the normal distribution N(μ, σ) of the absolute difference of n-1 timestamps 2 );

[0059] The third delay time determination unit is used to convert the normal distribution N(μ, σ 2 ) is determined as the absolute difference of the timestamp corresponding to the mean μ as the delay time T d .

[0060] Optionally, the above-mentioned Figure 2 The exemplary second determination module 203 may include a first difference calculation unit and a current latest waterline determination unit, wherein:

[0061] The first difference calculation unit is used to compare the timestamp of each data received in the current time window with the delay time T d Calculate the difference and get the difference value corresponding to each data received in the current time window;

[0062] The current latest water level determination unit is used to determine the largest difference among the differences as the current latest water level.

[0063] Optionally, the above-mentioned Figure 2 The exemplary third determination module 204 may include a second difference calculation unit and a triggering unit, wherein:

[0064] The second difference calculation unit is used to obtain the data D received in the current time window. current The current latest water level and delay time T d The difference between

[0065] The triggering unit is configured to trigger the current time window to start processing data received in the current time window if the difference value is equal to the right boundary of the current time window.

[0066] Optionally, the above-mentioned Figure 2 The exemplary apparatus may further include a first abandonment processing module or a second abandonment processing module, wherein:

[0067] The first abandonment processing module is used to current If other data is received later, the current The processing of other data received later;

[0068] The second abandonment processing module is used to receive data D within the current time window. current The current latest water level and delay time T d If the difference between them is greater than the right boundary of the current time window, the data D is abandoned. current processing.

[0069] From the description of the above technical solution, it can be seen that, on the one hand, due to the delay time T d It is calculated based on the absolute difference of n-1 timestamps. The absolute difference of n-1 timestamps is calculated strictly according to the timestamp between each piece of n out-of-order data and the last piece of non-out-of-order data in the previous event processing cycle, that is, the delay time T dThe acquisition of is objective, which provides an objective basis for the subsequent calculation of the latest water level, avoiding the existing technology relying on the experience of R&D personnel when determining the water level. Therefore, according to the current latest water level, the right boundary of the current time window and the delay time T d It is reasonable to determine whether to trigger the current time window to start processing the data received in the current time window; on the other hand, the latest watermark and whether to trigger the current time window to start processing the data received in the current time window are both based on the event processing cycle. Therefore, the response to data fluctuations, that is, data disorder, is relatively timely.

[0070] Figure 3 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 3 As shown, the computer device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a method for processing out-of-order data. When the processor 30 executes the computer program 32, the steps in the above-mentioned method for processing out-of-order data are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 2 The functions of the calculation module 201, the first determination module 202, the second determination module 203 and the third determination module 204 are shown.

[0071] Exemplarily, the computer program 32 of the method for processing out-of-order data mainly includes: calculating the absolute difference of the timestamp between each of n out-of-order data and the previous non-out-of-order data in the previous event processing cycle, obtaining n-1 timestamp absolute differences, where n is a natural number not less than 20; determining the delay time T according to the n-1 timestamp absolute differences d ; In the current event processing cycle, according to the timestamp and delay time T of the data received in the current time window d , determine the current latest water level; according to the current latest water level, the right boundary of the current time window and the delay time T d, determine whether to trigger the current time window to start processing the data received in the current time window. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments that can complete specific functions. The instruction segments are used to describe the execution process of the computer program 32 in the computer device 3. For example, the computer program 32 can be divided into the functions of the calculation module 201, the first determination module 202, the second determination module 203 and the third determination module 204 (modules in the virtual device). The specific functions of each module are as follows: the calculation module 201 is used to calculate the absolute difference in timestamps between each piece of n disordered data and the previous piece of non-disordered data in the previous event processing cycle, and obtain n-1 timestamp absolute differences, where n is a natural number not less than 20; the first determination module 202 is used to determine the delay time T according to the n-1 timestamp absolute differences. d The second determination module 203 is used to determine the timestamp and delay time T of the data received in the current time window in the current event processing cycle. d , determine the current latest waterline; the third determination module 204, for determining the current latest waterline, the right boundary of the current time window and the delay time T d , determines whether to trigger the current time window to start processing the data received in the current time window.

[0072] The computer device 3 may include but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation of computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output computer devices, network access computer devices, buses, etc.

[0073] The processor 30 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0074] The memory 31 can be an internal storage unit of the computer device 3, such as the hard disk or memory of the computer device 3. The memory 31 can also be an external storage device of the computer device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the computer device 3. Furthermore, the memory 31 can include both the internal storage unit of the computer device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the computer device. The memory 31 can also be used to temporarily store data that has been output or is about to be output.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0076] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0077] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0078] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0079] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0080] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0081] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-temporary computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program of the method for processing disordered data can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiments, that is, calculate the absolute difference in timestamps between each of the n disordered data and the previous non-disordered data in the previous event processing cycle, and obtain n-1 timestamp absolute differences, where n is a natural number not less than 20; according to the n-1 timestamp absolute differences, determine the delay time T d ; In the current event processing cycle, according to the timestamp and delay time T of the data received in the current time window d , determine the current latest water level; according to the current latest water level, the right boundary of the current time window and the delay time T d, determine whether to trigger the current time window to start processing the data received in the current time window. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. Non-transitory computer-readable media may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the non-transitory computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, non-transitory computer-readable media do not include electric carrier signals and telecommunication signals. The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0082] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation method of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of the present invention.

Claims

1. A method for processing out-of-order data, characterized in that: The method comprises: Calculate the absolute difference between the timestamps of each piece of n out-of-order data and the previous piece of non-out-of-order data in the previous event processing cycle, and obtain n-1 timestamp absolute differences, where n is a natural number not less than 20; Determine the delay time T based on the absolute difference of the n-1 timestamps d ; In the current event processing cycle, according to the timestamp of the data received in the current time window and the delay time T d , determine the current latest water level; According to the current latest waterline, the right boundary of the current time window and the delay time T d , determining whether to trigger the current time window to start processing the data received within the current time window; The delay time T is determined based on the absolute difference of the n-1 timestamps. d This includes any of the following methods: Method 1: Sort the absolute differences of the n-1 timestamps in order of size; The absolute difference of the timestamps corresponding to the specified sequence number is determined as the delay time T d , the specified sequence number is less than the maximum sequence number in the sort; Method 2: Sort the absolute differences of the n-1 timestamps in order of size; The absolute difference of the timestamp corresponding to the largest sort number in the sort is determined as the delay time T d ; Method 3: Obtain the normal distribution N(μ, σ) of the absolute differences of the n-1 timestamps 2 ); The normal distribution N(μ, σ 2 The absolute difference of the timestamps corresponding to the mean μ in the time series is determined as the delay time T d .

2. The method according to claim 1, wherein The timestamp of the data received in the current time window and the delay time T d , determine the current latest water level, including: The timestamp of each data received in the current time window is compared with the delay time T d Calculate the difference to obtain the difference value corresponding to each data received in the current time window; The largest difference among the differences is determined as the current latest water level line.

3. The method according to claim 1, wherein The current latest waterline, the right boundary of the current time window and the delay time T d , determining whether to trigger the current time window to start processing data received within the current time window, including: Obtain the data D received within the current time window current The current latest water level and the delay time T d The difference between If the difference is equal to the right boundary of the current time window, the current time window is triggered to start processing the data received in the current time window.

4. The method according to claim 3, wherein The method further comprises: If the data D is received current If other data is received after that, the current the processing of other data received subsequently; or If the difference is greater than the right boundary of the current time window, the data D is abandoned. current processing.

5. A device for processing out-of-order data, characterized in that: The device comprises: A calculation module is used to calculate the absolute difference between the timestamps of each piece of n out-of-order data and the previous piece of non-out-of-order data in the previous event processing cycle, and obtain n-1 absolute differences in the timestamps, where n is a natural number not less than 20; The first determining module is used to determine the delay time T according to the absolute difference of the n-1 timestamps. d ; The second determination module is used to determine, in the current event processing cycle, the timestamp of the data received in the current time window and the delay time T d , determine the current latest water level; The third determining module is used to determine the current latest waterline, the right boundary of the current time window and the delay time T d , determining whether to trigger the current time window to start processing the data received within the current time window; The first determining module includes a sorting unit and a first delay time determining unit, wherein the sorting unit is used to sort the n-1 timestamp absolute differences in order of size, and the first delay time determining unit is used to determine the timestamp absolute difference corresponding to the specified sequence number as the delay time T d , the specified sequence number is less than the maximum sequence number in the sort; or, The first determination module includes a sorting unit and a second delay time determination unit, wherein the sorting unit is used to sort the n-1 timestamp absolute differences in order of size, and the second delay time determination unit is used to determine the timestamp absolute difference corresponding to the largest sort number in the sort as the delay time T d ;or, The first determining module includes an obtaining unit and a third delay time determining unit, wherein the obtaining unit is used to obtain the normal distribution N (μ, σ) of the absolute differences of the n-1 timestamps. 2 ), the third delay time determination unit is used to: convert the normal distribution N(μ, σ 2 The absolute difference of the timestamps corresponding to the mean μ in the time series is determined as the delay time T d .

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for processing out-of-order data according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for processing out-of-order data according to any one of claims 1 to 4 are implemented.

Images