Data processing method, device and computer program product

Abstract

The disclosed embodiment discloses a data processing method, device and computer program product, the method comprising: in response to detecting the occurrence of an event in a rolling event detection window, performing a first aggregation process based on historical preset feature values ​​of similar events detected in the rolling event detection window, and obtaining an instantaneous window processing result corresponding to the event of this type; in response to the rolling event detection window reaching the rolling window duration, storing the latest instantaneous window processing result corresponding to the event of this type in the rolling event detection window as the window processing result corresponding to the event of this type in the rolling event detection window; in response to receiving a window processing result query command, obtaining the window processing result of the rolling event detection window corresponding to the window processing result query command, and performing a second aggregation process on the obtained window processing result. The technical solution can eliminate the redundancy of events involved in the calculation of aggregation indicators and reduce the use of computing resources and database resources.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and more particularly to a data processing method, device, and computer program product. Background Art

[0002] With the development of computing technology, the application of distributed streaming computing systems such as flink is becoming more and more extensive. For distributed streaming computing systems, sliding windows with a certain duration and sliding step length are usually used to calculate the aggregation index for the detection event, but the calculation method of the above sliding window in the prior art has the following disadvantages: due to the overlap and intersection of sliding windows, the same event may fall into different sliding windows and participate in the calculation of the aggregation index of the corresponding window, which will not only lead to a large number of aggregation index calculations, but also consume a lot of computing resources when a large number of events occur frequently, and also bring redundancy of events participating in the calculation of aggregation indexes, thereby affecting the accuracy of the final calculation of aggregation indexes; the above frequent calculation of window aggregation indexes will trigger frequent sliding window outputs, resulting in a higher database update TPS (Transaction Per Second), especially the sliding window calculation with a large event stream and a large time span requires a large amount of computing resources, which puts higher requirements on the database update performance. In the calculation of sliding window aggregation indexes of large-scale real-time event streams, it is also possible that the real-time calculation will be delayed for a long time due to insufficient resources, or even the calculation results cannot be obtained. Summary of the invention

[0003] Embodiments of the present disclosure provide a data processing method, device, and computer program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method.

[0005] Specifically, the data processing method includes:

[0006] In response to detecting the occurrence of an event in the rolling event detection window, performing a first aggregation process based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the same type;

[0007] In response to the rolling event detection window reaching the rolling window duration, storing the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type as the window processing result corresponding to the event of this type in the rolling event detection window;

[0008] In response to receiving a window processing result query command, the window processing results of one or more scroll event detection windows corresponding to the window processing result query command are obtained, and a second aggregation process is performed on the obtained window processing results to obtain an aggregated processing result.

[0009] In combination with the first aspect, in a first implementation of the first aspect, the embodiment of the present disclosure further includes:

[0010] One or more scroll event detection windows are set, wherein the times of different scroll event detection windows do not overlap.

[0011] In combination with the first aspect and the first implementation manner of the first aspect, in a second implementation manner of the first aspect of the embodiment of the present disclosure, the event corresponds to one of the scrolling event detection windows based on its occurrence time.

[0012] In combination with the first aspect and the above implementation of the first aspect, in a third implementation of the first aspect of the present disclosure, setting one or more scroll event detection windows includes:

[0013] Determine the number of scroll event detection windows;

[0014] Dividing the preset detection duration by the number of rolling event detection windows to obtain a rolling window duration;

[0015] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0016] In combination with the first aspect and the above implementation of the first aspect, in a fourth implementation of the first aspect of the present disclosure, setting one or more scroll event detection windows includes:

[0017] Obtaining a sliding window duration and a sliding window step length corresponding to the first aggregation process;

[0018] Determine a common divisor of the sliding window duration and the sliding window step length as the rolling window duration of the rolling event detection window;

[0019] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0020] In combination with the first aspect and the above implementation of the first aspect, in a fifth implementation of the first aspect of the present disclosure, the first aggregation processing is performed based on the historical preset feature values ​​of similar events historically detected in the rolling event detection window, including:

[0021] Obtaining preset characteristic values ​​of detected events;

[0022] Obtaining historical preset characteristic values ​​of similar events historically detected within the rolling event detection window;

[0023] A first aggregation process is performed on the preset feature value and the historical preset feature value.

[0024] In combination with the first aspect and the above-mentioned implementation method of the first aspect, in the sixth implementation method of the first aspect of the present disclosure, the window processing result query command carries the query start time, the query end time, the event type and the first aggregation processing type.

[0025] In combination with the first aspect and the above implementation manner of the first aspect, in a seventh implementation manner of the first aspect of the present disclosure, the acquiring window processing results of one or more scrolling event detection windows corresponding to the window processing result query command, and performing a second aggregation process on the acquired window processing results, includes:

[0026] Determine a start scroll event detection window and an end scroll event detection window according to the query start time and query end time;

[0027] Obtaining one or more window processing results corresponding to the event type and the first aggregation processing type from the starting scroll event detection window to the ending scroll event detection window;

[0028] A second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process is related to the first aggregation process.

[0029] In a second aspect, a data processing device is provided in an embodiment of the present disclosure.

[0030] Specifically, the data processing device includes:

[0031] A first execution module is configured to, in response to detecting the occurrence of an event in a rolling event detection window, perform a first aggregation process based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the same type;

[0032] a storage module configured to store, in response to the rolling event detection window reaching the rolling window duration, the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type as the window processing result corresponding to the rolling event detection window of this type;

[0033] The second execution module is configured to, in response to receiving a window processing result query command, obtain the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command, perform second aggregation processing on the obtained window processing results, and obtain an aggregation processing result.

[0034] In combination with the second aspect, in a first implementation of the second aspect of the embodiment of the present disclosure, the device may further include:

[0035] The setting module is configured to set one or more scroll event detection windows, wherein the times of different scroll event detection windows do not overlap.

[0036] In combination with the second aspect and the first implementation manner of the second aspect, in a second implementation manner of the second aspect of an embodiment of the present disclosure, the event corresponds to one of the scrolling event detection windows based on its occurrence time.

[0037] In combination with the second aspect and the above implementation of the second aspect, in a third implementation of the second aspect of the present disclosure, the setting module may be configured as follows:

[0038] Determine the number of rolling event detection windows;

[0039] Dividing the preset detection duration by the number of rolling event detection windows to obtain a rolling window duration;

[0040] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0041] In combination with the second aspect and the above implementation manner of the second aspect, in a fourth implementation manner of the second aspect of the present disclosure, the setting module may be configured as follows:

[0042] Obtaining a sliding window duration and a sliding window step length corresponding to the first aggregation process;

[0043] Determine a common divisor of the sliding window duration and the sliding window step length as the rolling window duration of the rolling event detection window;

[0044] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0045] In combination with the second aspect and the above implementation of the second aspect, in a fifth implementation of the second aspect of the present disclosure, the part that performs the first aggregation processing based on the historical preset feature values ​​of similar events historically detected in the rolling event detection window is configured as follows:

[0046] Obtaining preset characteristic values ​​of detected events;

[0047] Obtaining historical preset characteristic values ​​of similar events historically detected within the rolling event detection window;

[0048] A first aggregation process is performed on the preset feature value and the historical preset feature value.

[0049] In combination with the second aspect and the above-mentioned implementation method of the second aspect, in the sixth implementation method of the second aspect of the present disclosure, the window processing result query command carries the query start time, the query end time, the event type and the first aggregation processing type.

[0050] In combination with the second aspect and the foregoing implementation manner of the second aspect, in a seventh implementation manner of the second aspect of the present disclosure, the part of obtaining the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command and performing the second aggregation processing on the obtained window processing results may be configured as follows:

[0051] Determine a start scroll event detection window and an end scroll event detection window according to the query start time and query end time;

[0052] Obtaining one or more window processing results corresponding to the event type and the first aggregation processing type from the starting scroll event detection window to the ending scroll event detection window;

[0053] A second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process is related to the first aggregation process.

[0054] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions that support a data processing device to execute the above-mentioned data processing method, and the processor is configured to execute the computer instructions stored in the memory. The data processing device may also include a communication interface for the data processing device to communicate with other devices or a communication network.

[0055] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a data processing device, which includes computer instructions involved in the data processing device for executing the above-mentioned data processing method.

[0056] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method steps of the above-mentioned data processing method.

[0057] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0058] The above technical solution uses a rolling window instead of a sliding window to calculate event aggregation indicators. Since the time of different rolling windows does not overlap, a certain event only corresponds to one of the rolling windows according to its occurrence time. This technical solution can eliminate the redundancy of events involved in the calculation of aggregation indicators, improve the accuracy of aggregation indicator calculation, reduce the use of computing resources and database resources, avoid long delays in real-time calculations due to insufficient resources, or even the inability to obtain calculation results, thereby improving the calculation efficiency of event aggregation indicators and ensuring the stability of event aggregation indicator calculations.

[0059] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, purposes and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0061] Figure 1 A flowchart showing a data processing method according to an embodiment of the present disclosure;

[0062] Figure 2 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0063] Figure 3 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0064] Figure 4 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0066] In the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof existing or being added.

[0067] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0068] The technical solution provided by the embodiment of the present disclosure uses a rolling window instead of a sliding window to calculate event aggregation indicators. Since the time of different rolling windows does not overlap, a certain event only corresponds to one of the rolling windows according to its occurrence time. This technical solution can eliminate the redundancy of events involved in the calculation of aggregation indicators, improve the accuracy of aggregation indicator calculation, reduce the use of computing resources and database resources, avoid long delays in real-time calculations due to insufficient resources, or even the inability to obtain calculation results, thereby improving the calculation efficiency of event aggregation indicators and ensuring the stability of event aggregation indicator calculations.

[0069] Figure 1 A flow chart showing a data processing method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the data processing method includes the following steps S101-S103:

[0070] In step S101, in response to detecting the occurrence of an event in a rolling event detection window, a first aggregation process is performed based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the same type;

[0071] In step S102, in response to the rolling event detection window reaching the rolling window duration, the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type is stored as the window processing result corresponding to the event of this type in the rolling event detection window;

[0072] In step S103, in response to receiving a window processing result query command, the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command are obtained, and a second aggregation process is performed on the obtained window processing results to obtain an aggregated processing result.

[0073] As mentioned above, with the development of computing technology, the application of distributed streaming computing systems such as flink is becoming more and more extensive. For distributed streaming computing systems, sliding windows with a certain duration and sliding step length are usually used to calculate the aggregation index for the detection event, but the calculation method of the above sliding window in the prior art has the following disadvantages: due to the overlap and intersection of sliding windows, the same event may fall into different sliding windows and participate in the calculation of the aggregation index of the corresponding window, which will not only lead to a large number of aggregation index calculations, but also consume a lot of computing resources when a large number of events occur frequently, and also bring redundancy of events participating in the calculation of aggregation indexes, thereby affecting the accuracy of the final calculation of aggregation indexes; the above frequent calculation of window aggregation indexes will trigger frequent sliding window outputs, resulting in a higher database update TPS (Transaction Per Second), especially the sliding window calculation with a large event stream and a large time span requires a large amount of computing resources, which puts higher requirements on the database update performance. In the calculation of sliding window aggregation indexes of large-scale real-time event streams, it is also possible that the real-time calculation will be delayed for a long time due to insufficient resources, or even the calculation results cannot be obtained.

[0074] In view of the above problems, in this embodiment, a data processing method is proposed, which uses a rolling window instead of a sliding window to calculate event aggregation indicators. Since the time of different rolling windows does not overlap, a certain event only corresponds to one of the rolling windows according to its occurrence time. This technical solution can eliminate the redundancy of events involved in the calculation of aggregation indicators, improve the accuracy of aggregation indicator calculation, reduce the use of computing resources and database resources, avoid long delays in real-time calculations due to insufficient resources, or even the inability to obtain calculation results, thereby improving the calculation efficiency of event aggregation indicators and ensuring the stability of event aggregation indicator calculations.

[0075] In one embodiment of the present disclosure, the data processing method may be applicable to a computer, a computing device, an electronic device, a server, a server cluster, etc. that performs data processing.

[0076] In one embodiment of the present disclosure, the rolling event detection window refers to a time window with a preset rolling window duration for detecting the occurrence of an event. In this embodiment, unlike the sliding window, different rolling windows are independent of each other, and the corresponding detection times do not overlap, overlap or cross, and multiple sequential rolling windows are continuous and can cover a certain preset duration, that is, for adjacent rolling windows, the window end time of the previous rolling window is the window start time of the next rolling window, so for each event, it can be made to correspond to one and only one rolling window based on its occurrence time. For example, if N rolling windows are set within the preset duration T, or a sliding window with a preset duration T is split into N rolling windows, the rolling window duration of each rolling window is T / N, that is, N independent rolling windows are arranged in sequence, and the N rolling windows can completely cover the preset duration T.

[0077] In one embodiment of the present disclosure, the event refers to an event object that has an occurrence form, can be detected by the software system, and can record information such as the time of its occurrence. For example, the event can be a positioning request, data query request, etc. issued by a certain device. As mentioned above, different rolling windows are independent of each other, and the corresponding detection times do not overlap, overlap, or cross. Therefore, the occurrence time of an event will only fall into a certain rolling window, and will not fall into two or more rolling windows.

[0078] In one embodiment of the present disclosure, the similar events historically detected within the scrolling event detection window refer to all events that have been historically detected that belong to the same type as the event from the start time of a certain scrolling event detection window to the current time. For example, if the event detected in the scrolling event detection window in step S101 is a positioning request event, then the similar events historically detected within the scrolling event detection window refer to all positioning request events that have been historically detected from the start time of a certain scrolling event detection window to the current time.

[0079] In one embodiment of the present disclosure, the preset feature value refers to a pre-set feature value generated based on the event and involved in subsequent preset processing. The preset feature value can be, for example, the number of occurrences of the event, the amount of related data, the amount of memory occupied, etc. The historical preset feature value refers to the preset feature value corresponding to the historical detection event.

[0080] In one embodiment of the present disclosure, the first aggregation processing refers to a processing operation that is pre-set and performed based on the preset feature value. The first aggregation processing can be one or more types, that is, one or more aggregation processing operations can be performed based on the historical preset feature values ​​of the historical detection events in the rolling event detection window, and accordingly, one or more instantaneous window processing results corresponding to the aggregation processing operation can be obtained. The first aggregation processing can be, for example, processing operations such as summing, finding the maximum value, finding the minimum value, finding the average value, and counting. Among them, the first aggregation processing is triggered by the occurrence of an event, that is, once the occurrence of an event is detected in the rolling event detection window, the first aggregation processing is performed based on the preset feature values ​​of all detected events in the rolling event detection window. Therefore, each time the occurrence of an event is detected in the rolling event detection window, the first aggregation processing will be performed once, and a current instantaneous window processing result will be obtained. When the rolling event detection window reaches the rolling window duration, the latest instantaneous window processing result obtained at last is the window processing result corresponding to the rolling event detection window.

[0081] In one embodiment of the present disclosure, the window processing result query command refers to a command issued by a user, the system itself, or other resources for querying the window processing results of one or several scrolling event detection windows. In order to determine how many scrolling event detection windows there are to be queried, and specifically which windows, the window processing result query command may carry a query start time and a query end time. In order to determine the window processing result corresponding to the event to be queried, the window processing result query command may carry an event type. As mentioned above, the first aggregation processing may be one or more. In order to determine the accurate window processing result, the window processing result query command may also carry the type of the first aggregation processing.

[0082] In one embodiment of the present disclosure, the second aggregation processing refers to a pre-set processing operation related to the first aggregation processing, which is performed based on the window processing result. For example, if the first aggregation processing is a summation operation, the second aggregation processing can be the same as the first aggregation processing, which is also a summation operation, or it can be a processing operation different from the first aggregation processing but corresponding to the needs of the actual application. Similar to the first aggregation processing, the second aggregation processing can be processing operations such as summation, maximum value, minimum value, average value, counting, etc., which can be one or more. That is, one or more aggregation processing operations can be performed based on the window processing result, and accordingly, one or more aggregation processing results corresponding to the aggregation processing operation can be obtained.

[0083] In the above embodiment, when an event is detected in a scrolling event detection window, a first aggregation process is performed based on preset feature values ​​of all events of the same category as the event detected in the scrolling event detection window to obtain a current instantaneous window processing result corresponding to the event of this type; by analogy, if another event of the same type is detected, a first aggregation process is performed based on preset feature values ​​of all events of the same category detected in the scrolling event detection window to obtain an updated instantaneous window processing result, until the scrolling event detection window reaches the scrolling window length, that is, the scrolling event detection window ends, and the window processing result at this time is stored as the window processing result corresponding to the event of this type in the scrolling event detection window; after receiving a window processing result query command, the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command are obtained, and a second aggregation process is performed on the obtained window processing results to obtain an aggregate processing result.

[0084] In one embodiment of the present disclosure, the method may further include the following steps:

[0085] One or more scrolling event detection windows are set, wherein the scrolling event detection windows have a scrolling window duration, and the durations of different scrolling event detection windows do not overlap.

[0086] In this embodiment, when setting the rolling event detection window, the rolling window length can be determined according to the preset detection time. For example, the preset detection time can be set to 1 minute or 5 minutes, and then the number of rolling windows can be determined. The rolling window length can be determined based on the preset detection time and the number of rolling windows. The sliding window used in the previous aggregation calculation for a preset feature value can also be split into one or more rolling windows according to the needs of the actual application. For example, a sliding window with a duration of 1 hour and a sliding step of 5 minutes can be split into 12 rolling windows with a duration of 5 minutes, or 60 rolling windows with a duration of 1 minute. In this way, for each event, it can be made to correspond to one and only one rolling window based on its occurrence time.

[0087] When the rolling window duration is determined according to a preset detection duration, the step of setting one or more rolling event detection windows may include the following steps:

[0088] Determine the number of rolling event detection windows;

[0089] Dividing the preset detection duration by the number of rolling event detection windows to obtain a rolling window duration;

[0090] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0091] When the rolling window duration is determined according to the sliding window, the step of setting one or more rolling event detection windows may include the following steps:

[0092] Obtaining a sliding window duration and a sliding window step length corresponding to the first aggregation process;

[0093] Determine a common divisor of the sliding window duration and the sliding window step length as the rolling window duration of the rolling event detection window;

[0094] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0095] In this implementation, when there is more than one common divisor of the sliding window duration and the sliding window step length, one of the common divisors can be selected as the rolling window duration of the rolling event detection window according to the needs of the actual application.

[0096] In one embodiment of the present disclosure, the step of performing the first aggregation processing based on the historical preset feature values ​​of the same type of events historically detected in the rolling event detection window in step S101 may include the following steps:

[0097] Obtaining preset characteristic values ​​of detected events;

[0098] Obtaining historical preset characteristic values ​​of similar events historically detected within the rolling event detection window;

[0099] A first aggregation process is performed on the preset feature value and the historical preset feature value.

[0100] In this embodiment, when performing the first aggregation process based on the preset feature values ​​of similar events detected historically in the rolling event detection window, it is necessary to calculate the preset feature values ​​of all similar events detected in the rolling event detection window, and then perform the first aggregation process based on the calculated preset feature values. For example, the preset feature value of the most recently detected event can be obtained first; then the preset feature value of the previously detected similar events in the rolling event detection window can be obtained; and finally, the first aggregation process can be performed on all the obtained preset feature values.

[0101] In one embodiment of the present disclosure, the step of obtaining the window processing results of one or more scroll event detection windows corresponding to the window processing result query command in step S103, and performing the second aggregation processing on the obtained window processing results may include the following steps:

[0102] Determine a start scroll event detection window and an end scroll event detection window according to the query start time and query end time;

[0103] Obtaining one or more window processing results corresponding to the event type and the first aggregation processing type from the starting scroll event detection window to the ending scroll event detection window;

[0104] A second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process corresponds to the first aggregation process.

[0105] As mentioned above, in order to determine how many rolling event detection windows to be queried, and which windows they are, the window processing result query command may carry the query start time and query end time. In order to determine the accurate window processing result, the window processing result query command may also carry the event type and the type of the first aggregation processing. It is also mentioned above that the aggregation calculation for a preset feature value in the prior art uses a sliding window with a certain duration. Therefore, in this embodiment, the query start time and query end time may be determined based on the duration and sliding step of the sliding window used for the aggregation calculation for a preset feature value. For example, if the duration of the sliding window is 1 hour, the sliding step is 5 minutes, and the current time is 11 a.m., the query start time may be set to 10 a.m., and the query end time may be set to 11 a.m. The next query start time may be set to 10 a.m. and the query end time may be set to 11 a.m., and the query start time may be set to 10 a.m. and 11 a.m., and so on.

[0106] After determining the query start time and the query end time, the starting scrolling event detection window and the ending scrolling event detection window can be determined. For example, if the query start time is 10 o'clock, the query end time is 11 o'clock, and the scrolling window length of the scrolling event detection window is 5 minutes, then the starting scrolling event detection window is the scrolling event detection window from 10 o'clock to 10:05, and the ending scrolling event detection window is the scrolling event detection window from 10:55 to 11 o'clock. The number of query scrolling event detection windows is 12.

[0107] Then, multiple window processing results corresponding to the event type and the first aggregation processing type are obtained from the starting scrolling event detection window to the ending scrolling event detection window. For example, if there are 12 query scrolling event detection windows from the starting scrolling event detection window to the ending scrolling event detection window, but there are only 10 scrolling event detection windows corresponding to the event type and the first aggregation processing type, then there are only 10 corresponding window processing results.

[0108] Finally, a second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process is related to the first aggregation process.

[0109] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0110] Figure 2 FIG. 1 is a block diagram showing a data processing device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 2 As shown, the data processing device includes:

[0111] The first execution module 201 is configured to, in response to detecting the occurrence of an event in the rolling event detection window, perform a first aggregation process based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the type;

[0112] The storage module 202 is configured to store the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type as the window processing result corresponding to the rolling event detection window of this type of event in response to the rolling event detection window reaching the rolling window duration;

[0113] The second execution module 203 is configured to, in response to receiving a window processing result query command, obtain the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command, and perform a second aggregation processing on the obtained window processing results to obtain an aggregation processing result.

[0114] As mentioned above, with the development of computing technology, the application of distributed streaming computing systems such as flink is becoming more and more extensive. For distributed streaming computing systems, sliding windows with a certain duration and sliding step length are usually used to calculate the aggregation index for the detection event, but the calculation method of the above sliding window in the prior art has the following disadvantages: due to the overlap and intersection of sliding windows, the same event may fall into different sliding windows and participate in the calculation of the aggregation index of the corresponding window, which will not only lead to a large number of aggregation index calculations, but also consume a lot of computing resources when a large number of events occur frequently, and also bring redundancy of events participating in the calculation of aggregation indexes, thereby affecting the accuracy of the final calculation of aggregation indexes; the above frequent calculation of window aggregation indexes will trigger frequent sliding window outputs, resulting in a higher database update TPS (Transaction Per Second), especially the sliding window calculation with a large event stream and a large time span requires a large amount of computing resources, which puts higher requirements on the database update performance. In the calculation of sliding window aggregation indexes of large-scale real-time event streams, it is also possible that the real-time calculation will be delayed for a long time due to insufficient resources, or even the calculation results cannot be obtained.

[0115] In view of the above problems, in this embodiment, a data processing device is proposed, which uses a rolling window instead of a sliding window to calculate event aggregation indicators. Since the time of different rolling windows does not overlap, a certain event only corresponds to one of the rolling windows according to its occurrence time. This technical solution can eliminate the redundancy of events involved in the calculation of aggregation indicators, improve the accuracy of aggregation indicator calculation, reduce the use of computing resources and database resources, avoid long delays in real-time calculations due to insufficient resources, or even the inability to obtain calculation results, thereby improving the calculation efficiency of event aggregation indicators and ensuring the stability of event aggregation indicator calculations.

[0116] In one embodiment of the present disclosure, the data processing apparatus may be implemented as a computer, a computing device, an electronic device, a server, a server cluster, etc. that performs data processing.

[0117] In one embodiment of the present disclosure, the rolling event detection window refers to a time window with a preset rolling window duration for detecting the occurrence of an event. In this embodiment, unlike the sliding window, different rolling windows are independent of each other, and the corresponding detection times do not overlap, overlap or cross, and multiple sequential rolling windows are continuous and can cover a certain preset duration, that is, for adjacent rolling windows, the window end time of the previous rolling window is the window start time of the next rolling window, so for each event, it can be made to correspond to one and only one rolling window based on its occurrence time. For example, if N rolling windows are set within the preset duration T, or a sliding window with a preset duration T is split into N rolling windows, the rolling window duration of each rolling window is T / N, that is, N independent rolling windows are arranged in sequence, and the N rolling windows can completely cover the preset duration T.

[0118] In one embodiment of the present disclosure, the event refers to an event object that has an occurrence form, can be detected by the software system, and can record information such as the time of its occurrence. For example, the event can be a positioning request, data query request, etc. issued by a certain device. As mentioned above, different rolling windows are independent of each other, and the corresponding detection times do not overlap, overlap, or cross. Therefore, the occurrence time of an event will only fall into a certain rolling window, and will not fall into two or more rolling windows.

[0119] In one embodiment of the present disclosure, the similar events historically detected within the scrolling event detection window refer to all events that have been historically detected and belong to the same type as the event from the start time of a certain scrolling event detection window to the current time. For example, if the event detected in the scrolling event detection window in the first execution module is a positioning request event, then the similar events historically detected within the scrolling event detection window refer to all positioning request events that have been historically detected from the start time of a certain scrolling event detection window to the current time.

[0120] In one embodiment of the present disclosure, the preset feature value refers to a pre-set feature value generated based on the event and involved in subsequent preset processing. The preset feature value can be, for example, the number of occurrences of the event, the amount of related data, the amount of memory occupied, etc. The historical preset feature value refers to the preset feature value corresponding to the historical detection event.

[0121] In one embodiment of the present disclosure, the first aggregation processing refers to a processing operation that is pre-set and performed based on the preset feature value. The first aggregation processing can be one or more types, that is, one or more aggregation processing operations can be performed based on the historical preset feature values ​​of the historical detection events in the rolling event detection window, and accordingly, one or more instantaneous window processing results corresponding to the aggregation processing operation can be obtained. The first aggregation processing can be, for example, processing operations such as summing, finding the maximum value, finding the minimum value, finding the average value, and counting. Among them, the first aggregation processing is triggered by the occurrence of an event, that is, once the occurrence of an event is detected in the rolling event detection window, the first aggregation processing is performed based on the preset feature values ​​of all detected events in the rolling event detection window. Therefore, each time the occurrence of an event is detected in the rolling event detection window, the first aggregation processing will be performed once, and a current instantaneous window processing result will be obtained. When the rolling event detection window reaches the rolling window duration, the latest instantaneous window processing result obtained at last is the window processing result corresponding to the rolling event detection window.

[0122] In one embodiment of the present disclosure, the window processing result query command refers to a command issued by a user, the system itself, or other resources for querying the window processing results of one or several scrolling event detection windows. In order to determine how many scrolling event detection windows there are to be queried, and specifically which windows, the window processing result query command may carry a query start time and a query end time. In order to determine the window processing result corresponding to the event to be queried, the window processing result query command may carry an event type. As mentioned above, the first aggregation processing may be one or more. In order to determine the accurate window processing result, the window processing result query command may also carry the type of the first aggregation processing.

[0123] In one embodiment of the present disclosure, the second aggregation processing refers to a pre-set processing operation related to the first aggregation processing, which is performed based on the window processing result. For example, if the first aggregation processing is a summation operation, the second aggregation processing can be the same as the first aggregation processing, which is also a summation operation, or it can be a processing operation different from the first aggregation processing but corresponding to the needs of the actual application. Similar to the first aggregation processing, the second aggregation processing can be processing operations such as summation, maximum value, minimum value, average value, counting, etc., which can be one or more. That is, one or more aggregation processing operations can be performed based on the window processing result, and accordingly, one or more aggregation processing results corresponding to the aggregation processing operation can be obtained.

[0124] In the above embodiment, when an event is detected in a scrolling event detection window, a first aggregation process is performed based on preset feature values ​​of all events of the same category as the event detected in the scrolling event detection window to obtain a current instantaneous window processing result corresponding to the event of this type; by analogy, if another event of the same type is detected, a first aggregation process is performed based on preset feature values ​​of all events of the same category detected in the scrolling event detection window to obtain an updated instantaneous window processing result, until the scrolling event detection window reaches the scrolling window length, that is, the scrolling event detection window ends, and the window processing result at this time is stored as the window processing result corresponding to the event of this type in the scrolling event detection window; after receiving a window processing result query command, the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command are obtained, and a second aggregation process is performed on the obtained window processing results to obtain an aggregate processing result.

[0125] In one embodiment of the present disclosure, the device may further include:

[0126] The setting module is configured to set one or more scrolling event detection windows, wherein the scrolling event detection window has a scrolling window duration, and the durations of different scrolling event detection windows do not overlap.

[0127] In this embodiment, when setting the rolling event detection window, the rolling window length can be determined according to the preset detection time. For example, the preset detection time can be set to 1 minute or 5 minutes, and then the number of rolling windows can be determined. The rolling window length can be determined based on the preset detection time and the number of rolling windows. The sliding window used in the previous aggregation calculation for a preset feature value can also be split into one or more rolling windows according to the needs of the actual application. For example, a sliding window with a duration of 1 hour and a sliding step of 5 minutes can be split into 12 rolling windows with a duration of 5 minutes, or 60 rolling windows with a duration of 1 minute. In this way, for each event, it can be made to correspond to one and only one rolling window based on its occurrence time.

[0128] When the rolling window duration is determined according to a preset detection duration, the setting module may be configured as follows:

[0129] Determine the number of rolling event detection windows;

[0130] Dividing the preset detection duration by the number of rolling event detection windows to obtain a rolling window duration;

[0131] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0132] When the rolling window duration is determined according to the sliding window, the setting module may be configured as follows:

[0133] Obtaining a sliding window duration and a sliding window step length corresponding to the first aggregation process;

[0134] Determine a common divisor of the sliding window duration and the sliding window step length as the rolling window duration of the rolling event detection window;

[0135] The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

[0136] In this implementation, when there is more than one common divisor of the sliding window duration and the sliding window step length, one of the common divisors can be selected as the rolling window duration of the rolling event detection window according to the needs of the actual application.

[0137] In one embodiment of the present disclosure, the part of performing the first aggregation processing based on the historical preset feature values ​​of the same type of events historically detected in the rolling event detection window is configured as follows:

[0138] Obtaining preset characteristic values ​​of detected events;

[0139] Obtaining historical preset characteristic values ​​of similar events historically detected within the rolling event detection window;

[0140] A first aggregation process is performed on the preset feature value and the historical preset feature value.

[0141] In this embodiment, when performing the first aggregation process based on the preset feature values ​​of similar events detected historically in the rolling event detection window, it is necessary to calculate the preset feature values ​​of all similar events detected in the rolling event detection window, and then perform the first aggregation process based on the calculated preset feature values. For example, the preset feature value of the most recently detected event may be obtained first; then the preset feature value of the previously detected similar events in the rolling event detection window may be obtained; and finally, the first aggregation process may be performed on all the obtained preset feature values.

[0142] In an embodiment of the present disclosure, the part of acquiring the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command and performing the second aggregation processing on the acquired window processing results may be configured as follows:

[0143] Determine a start scroll event detection window and an end scroll event detection window according to the query start time and query end time;

[0144] Obtaining one or more window processing results corresponding to the event type and the first aggregation processing type from the starting scroll event detection window to the ending scroll event detection window;

[0145] A second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process corresponds to the first aggregation process.

[0146] As mentioned above, in order to determine how many rolling event detection windows to be queried, and which windows they are, the window processing result query command may carry the query start time and query end time. In order to determine the accurate window processing result, the window processing result query command may also carry the event type and the type of the first aggregation processing. It is also mentioned above that the aggregation calculation for a preset feature value in the prior art uses a sliding window with a certain duration. Therefore, in this embodiment, the query start time and query end time may be determined based on the duration and sliding step of the sliding window used for the aggregation calculation for a preset feature value. For example, if the duration of the sliding window is 1 hour, the sliding step is 5 minutes, and the current time is 11 a.m., the query start time may be set to 10 a.m., and the query end time may be set to 11 a.m. The next query start time may be set to 10 a.m. and the query end time may be set to 11 a.m., and the query start time may be set to 10 a.m. and 11 a.m., and so on.

[0147] After determining the query start time and the query end time, the starting scrolling event detection window and the ending scrolling event detection window can be determined. For example, if the query start time is 10 o'clock, the query end time is 11 o'clock, and the scrolling window length of the scrolling event detection window is 5 minutes, then the starting scrolling event detection window is the scrolling event detection window from 10 o'clock to 10:05, and the ending scrolling event detection window is the scrolling event detection window from 10:55 to 11 o'clock. The number of query scrolling event detection windows is 12.

[0148] Then, multiple window processing results corresponding to the event type and the first aggregation processing type are obtained from the starting scrolling event detection window to the ending scrolling event detection window. For example, if there are 12 query scrolling event detection windows from the starting scrolling event detection window to the ending scrolling event detection window, but there are only 10 scrolling event detection windows corresponding to the event type and the first aggregation processing type, then there are only 10 corresponding window processing results.

[0149] Finally, a second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process is related to the first aggregation process.

[0150] The present disclosure also discloses an electronic device, Figure 3A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 3 As shown, the electronic device 300 includes a memory 301 and a processor 302; wherein,

[0151] The memory 301 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 302 to implement the above method steps.

[0152] Figure 4 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0153] like Figure 4 As shown, the computer system 400 includes a processing unit 401, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0154] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. The drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 410 as needed, so that the computer program read therefrom is installed into the storage section 408 as needed. Among them, the processing unit 401 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0155] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing the data processing method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411.

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be arranged in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0158] As another aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A data processing method, comprising: In response to detecting the occurrence of an event in the rolling event detection window, performing a first aggregation process based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the same type, wherein the historical preset feature value refers to a preset feature value corresponding to a historical detection event, and the first aggregation process refers to a pre-set processing operation performed based on the preset feature value; In response to the rolling event detection window reaching the rolling window duration, storing the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type as the window processing result corresponding to the event of this type in the rolling event detection window; In response to receiving a window processing result query command, the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command are obtained, and a second aggregation processing is performed on the obtained window processing results to obtain an aggregation processing result, wherein the second aggregation processing refers to a pre-set processing operation related to the first aggregation processing and performed based on the window processing result.

2. The method according to claim 1, further comprising: One or more scroll event detection windows are set, wherein the times of different scroll event detection windows do not overlap. The method according to claim 2 , wherein the event corresponds to one of the rolling event detection windows based on its occurrence time.

4. The method according to claim 2, wherein setting one or more scroll event detection windows comprises: Determine the number of rolling event detection windows; Divide the preset detection duration by the number of rolling event detection windows to obtain the rolling window duration; The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

5. The method according to claim 2, wherein setting one or more scroll event detection windows comprises: Obtaining a sliding window duration and a sliding window step length corresponding to the first aggregation process; Determine a common divisor of the sliding window duration and the sliding window step length as the rolling window duration of the rolling event detection window; The end time of the previous scrolling event detection window is used as the start time of the next scrolling event detection window, and the next scrolling event detection window is set based on the scrolling window duration.

6. The method according to any one of claims 1 to 5, wherein the first aggregation process is performed based on historical preset feature values ​​of similar events historically detected within the rolling event detection window, comprising: Obtaining preset characteristic values ​​of detected events; Obtaining historical preset characteristic values ​​of similar events historically detected within the rolling event detection window; A first aggregation process is performed on the preset feature value and the historical preset feature value.

7. According to the method according to any one of claims 1-5, the window processing result query command carries the query start time, the query end time, the event type and the first aggregation processing type.

8. According to the method of claim 7, the acquiring window processing results of one or more scroll event detection windows corresponding to the window processing result query command, and performing a second aggregation process on the acquired window processing results, comprises: Determine a start scroll event detection window and an end scroll event detection window according to the query start time and query end time; Obtaining one or more window processing results corresponding to the event type and the first aggregation processing type from the starting scroll event detection window to the ending scroll event detection window; A second aggregation process is performed based on the one or more window processing results, wherein the second aggregation process is related to the first aggregation process.

9. A data processing device, comprising: A first execution module is configured to, in response to detecting the occurrence of an event in a rolling event detection window, perform a first aggregation process based on historical preset feature values ​​of similar events historically detected in the rolling event detection window to obtain an instantaneous window processing result corresponding to the event of the same type, wherein the historical preset feature value refers to a preset feature value corresponding to a historical detection event, and the first aggregation process refers to a pre-set processing operation performed based on the preset feature value; a storage module configured to store, in response to the rolling event detection window reaching the rolling window duration, the latest instantaneous window processing result of the rolling event detection window corresponding to the event of this type as the window processing result corresponding to the rolling event detection window of this type; The second execution module is configured to, in response to receiving a window processing result query command, obtain the window processing results of one or more scrolling event detection windows corresponding to the window processing result query command, perform a second aggregation processing on the obtained window processing results, and obtain an aggregation processing result, wherein the second aggregation processing refers to a pre-set processing operation related to the first aggregation processing and performed based on the window processing result.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the method steps described in any one of claims 1 to 8 are implemented.

Images