Streaming data processing method and device based on Flink window, equipment and storage medium

Abstract

The invention discloses a streaming data processing method, device and equipment based on an Flink window and a storage medium, and relates to the technical field of streaming data processing.The method comprises the steps that a task starting time point and task duration are determined according to task demand information; when stream data corresponding to the task starting time point is received, acquiring an event time point of the stream data, and registering an expiration time point timer of the stream data according to the event time point and the task duration; and when the time triggering condition of the expiration time point timer is met, performing window operation on the data in the window of the window where the flow data is located, and obtaining a task output result. According to the method and the device, the window operation is triggered based on the event time point and the task duration, so that the time sequence consistency of the Flink window calculation result can be kept under the condition of delayed arrival of the stream data, and the real-time high-precision requirement of the task output result is met.

Description

Technical Field

[0001] The present application relates to the technical field of stream data processing, and in particular to a stream data processing method, apparatus, device and storage medium based on Flink window. Background Art

[0002] With the expansion of online financial services, the popularization of electronic payments, and the large number of online transactions, financial institutions are facing increasing risk control pressure. In order to ensure the security of financial services, a big data real-time risk control system is required to provide real-time risk control capabilities. This real-time risk control system is usually a stream data processing system, which can process the stream data generated in financial services in real time or near real time to meet the needs of different risk control decision-making tasks.

[0003] Existing stream data processing methods use two independent dimensions, processing time and event time, to process stream data arriving within a window. However, due to the different sources of stream data in financial services and the differences in the time at which different stream data arrive at the window, that is, the stream data arrives at the system window late, and there may be a large difference between the processing time and the event time, which makes it difficult for the window-based stream data calculation results to meet the real-time and high-precision calculation requirements of risk control decision-making tasks. Summary of the invention

[0004] The main purpose of this application is to provide a stream data processing method, device, equipment and storage medium based on Flink window, aiming to solve the technical problem of low accuracy of stream data calculation results based on system window when stream data arrives late.

[0005] To achieve the above objectives, the present application proposes a stream data processing method based on Flink window, the method comprising:

[0006] Determine the task start time and duration based on task requirement information;

[0007] When receiving the stream data corresponding to the task start time point, obtaining the event time point of the stream data, and registering the expiration time point timer of the stream data according to the event time point and the task duration;

[0008] When the time trigger condition of the expiration time point timer is met, a window operation is performed on the data in the window where the stream data is located to obtain a task output result.

[0009] In one embodiment, when receiving the stream data corresponding to the task start time point, the step of obtaining the event time point of the stream data, and registering the expiration time point timer of the stream data according to the event time point and the task duration includes:

[0010] Determine the task start time point, and obtain the event time point of the stream data when the window receives the stream data corresponding to the task time point;

[0011] Determine the expiration time point of the stream data according to the event time point and the task duration;

[0012] An expiration time point timer for the stream data is generated according to the expiration time point.

[0013] In one embodiment, after the step of determining the task start time point, the method further includes:

[0014] When the window receives the stream data corresponding to the task start time point, the data in the window is updated;

[0015] Perform window operations based on the update results to obtain window calculation results.

[0016] In one embodiment, the window includes original data of the window, and after the step of generating an expiration time point timer of the stream data according to the expiration time point, the step further includes:

[0017] When the current time point satisfies the time trigger condition of the expiration time point timer of the original data in the window, the data in the window is updated to remove the expired data in the original data in the window;

[0018] Perform window operations based on the update results to obtain window calculation results.

[0019] In one embodiment, when the time trigger condition of the expiration time point timer is met, the step of performing a window operation on the data in the window where the stream data is located to obtain the task output result includes:

[0020] When the current time point satisfies the time trigger condition of the expiration time point timer of the stream data, a window operation is performed on the data in the window where the stream data is located, and the window calculation result is updated to obtain a window withdrawal calculation result, wherein the data in the window includes the stream data and the non-expired data in the original data of the window;

[0021] The window withdrawal calculation result is determined as the task output result.

[0022] In one embodiment, after the step of determining the window withdrawal calculation result as the task output result, the method further includes:

[0023] Updating the data in the window to withdraw the stream data, the stream data being expired data;

[0024] A window operation is performed based on the update result to obtain a window calculation result, where the window calculation result is zero.

[0025] In one embodiment, the method further comprises:

[0026] The window calculation result or the task output result is stored in an external system through a Sink operator;

[0027] The TTL service based on the external system determines the zeroing window time point of the task output result according to the task duration;

[0028] When the current time point meets the zeroing window time point, a zeroing operation is triggered to clear the task output result.

[0029] In addition, to achieve the above purpose, the present application also proposes a stream data processing device based on Flink window, the device comprising:

[0030] The demand configuration module is used to determine the task start time and task duration according to the task demand information;

[0031] An expiration point registration module is used to obtain the event time point of the stream data when receiving the stream data corresponding to the task start time point, and register the expiration time point timer of the stream data according to the event time point and the task duration;

[0032] The result output module is used to perform a window operation on the data in the window where the stream data is located to obtain a task output result when the time trigger condition of the expiration time point timer is met.

[0033] In addition, to achieve the above-mentioned objectives, the present application also proposes a Flink window-based stream data processing device, the device comprising: a memory, a processor, and a Flink window-based stream data processing program stored in the memory and executable on the processor, the Flink window-based stream data processing program being configured to implement the steps of the Flink window-based stream data processing method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a Flink window-based stream data processing program is stored. When the Flink window-based stream data processing program is executed by a processor, the steps of the Flink window-based stream data processing method described above are implemented.

[0035] The present application discloses a method for processing stream data based on Flink windows, which determines the task start time point and task duration according to task requirement information; when receiving the stream data corresponding to the task start time point, obtains the event time point of the stream data, and registers the expiration time point timer of the stream data according to the event time point and the task duration; when the time trigger condition of the expiration time point timer is met, performs a window operation on the data in the window where the stream data is located to obtain the task output result. Since the present application triggers the Flink window operation based on the event time point and the task duration, it can maintain the temporal consistency of the Flink window calculation results when the stream data arrives late, thereby meeting the real-time and high-precision requirements of the task output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 This is a flow chart of the first embodiment of the stream data processing method based on Flink window of this application;

[0039] Figure 2 This is a flow chart of the second embodiment of the stream data processing method based on Flink window of this application;

[0040] Figure 3 This is the Flink window calculation logic diagram for this application;

[0041] Figure 4 This is a flow chart of the third embodiment of the stream data processing method based on Flink window of the present application;

[0042] Figure 5 Activate the calculation implementation graph for the existing data window withdrawal;

[0043] Figure 6 This is a structural block diagram of the first embodiment of the stream data processing device based on Flink window of the present application;

[0044] Figure 7 This is a structural diagram of a stream data processing device based on Flink window in this application.

[0045] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The present application embodiment provides a stream data processing method based on Flink window, refer to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the stream data processing method based on Flink window of the present application. In this embodiment, the method includes: steps S10 to S30:

[0049] Step S10: Determine the task start time and task duration according to the task requirement information.

[0050] It should be noted that the present application can be applied in financial risk control scenarios, and can also be applied to scenarios such as stock market analysis and traffic analysis that require the application of a stream data processing system for stream data processing and analysis. The executor of the method of this embodiment can be a computer electronic device that has functions such as data processing, program running, and network communication, and can be connected to a stream data processing system, such as a server, a computer, a mobile phone, etc. The following can take a stream data processing device based on a Flink window connected to a stream data processing system as an example to explain the various embodiments of the present application.

[0051] It is understandable that the stream data processing system in the present application can be built based on the Flink framework. Flink is a distributed stream processing framework for performing stateful computations on unbounded and bounded data streams. Flink predefines multiple window operators (processing windows) for performing window computations on data streams entering the system to meet the data processing requirements of different tasks.

[0052] It should be understood that the task requirement information may include the user's requirement for calculating the flow variables of the flow data. For example, if the task requirement information is: transaction statistics task: the user needs to count the transaction data of a certain customer in the last 24 hours. Then the time when the processing window receives the latest flow data can be determined as the task start time point T, and the task duration can be determined as 24 hours.

[0053] Step S20: upon receiving the stream data corresponding to the task start time point, obtaining the event time point of the stream data, and registering an expiration time point timer of the stream data according to the event time point and the task duration.

[0054] It should be noted that in Flink, when data arrives at the processing window, a timer is registered through the TimerService, triggering the onTimer function to process the data in the window. The processing steps include state cleanup, withdrawal of expired data, accumulator calculation, and result output.

[0055] It should be understood that the Over window is a type of Flink window operator, which can perform aggregate calculations on a group of data based on a row or a row of data (timestamp). In the Over window aggregation operator, the timer registration and triggering process only occurs when new data arrives or all data expires. In order to trigger the update of the calculation result output when each piece of data expires, the quitTsState state (window exit state identifier) ​​can be added to the aggregation operator processing class. By introducing the annotation event time field, the current job window time span is calculated, and the expiration time point timer is registered at the same time when processing the data. Modify the onTimer function, add the judgment condition, and trigger the window exit mechanism on demand when the time is met at the current time point to trigger the window calculation, ensure on-demand calculation, and produce accurate calculation results.

[0056] It is understandable that incremental on-demand computing services can be implemented by modifying the underlying trigger mechanism of Flink, and then customized on-demand incremental computing of Flink Over windows can be implemented, and customized annotation event time fields and related configurations for different task scenarios can be predefined, which are compatible with general Flink SQL task-related configurations, so that users can better focus on the tasks themselves. Since a column of the data source to which the data stream belongs can be specified as the window-back time base point through annotations, it can adapt to general Flink data source fields without technical intervention in development, which is convenient for users to configure and use.

[0057] It should be noted that in the existing method, due to the delayed arrival of the streaming data entering the window, the expiration time point determined based on the data processing time and the window running time is delayed accordingly, that is, the expired data cannot be processed and the calculation results cannot be updated, resulting in the window calculation results being too large, which in turn affects the accuracy of the task output results.

[0058] Based on this, in this embodiment, in order to avoid the impact of delayed data arrival, event time is used instead of processing time when registering the expiration time point timer. Since the event time is the time when the data is actually generated, it is possible to maintain the temporal consistency of the final calculation results of the Flink processing window regardless of delayed data arrival or data arrival out of order.

[0059] In order to specifically explain how to register the expiration time point timer of the flow data arriving at the processing window, step S20 includes: steps S201 to S203:

[0060] Step S201: determine the task start time point, and obtain the event time point of the stream data when the window receives the stream data corresponding to the task time point.

[0061] It is understandable that the stream data corresponding to the starting time point of the task can be the latest stream data received by the above processing window. Due to data delay, that is, the processing time point when the stream data arrives at the processing window may be significantly different from the time point when it is actually generated, so the time point when the stream data is actually generated can be determined first.

[0062] Step S202: Determine the expiration time point of the stream data according to the event time point and the task duration.

[0063] Step S203: Generate an expiration time point timer for the stream data according to the expiration time point.

[0064] It should be understood that the expiration time point of the stream data may be determined by “expiration time point = event time point + window size”, wherein the window size may be the task duration determined by the task requirement.

[0065] In a specific implementation, when receiving the latest stream data, the processing system can obtain the event time point of the stream data, and register an expiration time point timer according to the event time point and the task duration.

[0066] Step S30: When the time trigger condition of the expiration time point timer is met, a window operation is performed on the data in the window where the stream data is located to obtain a task output result.

[0067] It should be understood that as time goes by, after the start time point of the aforementioned task, the data that previously belonged to the processing window reaches its corresponding expiration time point. After exceeding this expiration time point, the number of data in the window will change, and window calculation can also be triggered at this time.

[0068] It can be understood that until the time passes to the current time point and reaches the expiration time point of the above-mentioned stream data, the above-mentioned stream data will be regarded as expired data and withdrawn from the window. At this time, it can be regarded as satisfying the time required for the task to trigger the window exit mechanism on demand, and then the window aggregation calculation corresponding to the task is performed on the window data in the window where the stream data is located, and the obtained window calculation result (window exit calculation result) is used as the task output result.

[0069] This embodiment determines the task start time point and task duration according to the task requirement information; when receiving the stream data corresponding to the task start time point, obtains the event time point of the stream data, and registers the expiration time point timer of the stream data according to the event time point and the task duration; when the time trigger condition of the expiration time point timer is met, performs a window operation on the data in the window where the stream data is located to obtain the task output result. Since this embodiment can trigger the Flink window operation based on the event time point and the task duration, it can maintain the timing consistency of the Flink window calculation results when the stream data arrives late, thereby meeting the real-time and high-precision requirements of the task output results.

[0070] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the stream data processing method based on Flink window of the present application.

[0071] In this embodiment, since the calculation result update is triggered when each piece of data flows into the processing window, after step S201, steps S2011 to S2012 are also included:

[0072] Step S2011: When the window receives the stream data corresponding to the task start time point, the data in the window is updated.

[0073] It should be understood that when the latest stream data is received in the Flink window of the processing system, the number of original data in the window can be updated, that is, the value of the accumulator corresponding to the window can be updated or calculated.

[0074] Step S2012: Perform window operations based on the update results to obtain window calculation results.

[0075] It should be understood that the data in the window at this time includes the latest stream data and the original data in the window, and then the window aggregation calculation corresponding to the task is performed based on the data in the window to obtain the window calculation result. This is to facilitate the subsequent comparison of the window calculation result with the final task output result obtained to meet different task requirements.

[0076] Furthermore, considering that as time goes by, the original data in the window successively reaches the corresponding expiration time and becomes expired data, therefore, after step S203, the following steps are further included: steps S2031 to S2032:

[0077] Step S2031: when the current time point satisfies the time trigger condition of the expiration time point timer of the original data in the window, the data in the window is updated to remove the expired data in the original data in the window.

[0078] Step S2032: Perform window operations based on the update results to obtain window calculation results.

[0079] It should be understood that as time goes by, when the current time point reaches the expiration time point of the original data in the window, the original data in the window can be regarded as expired data, and the expired data can be withdrawn from the window to obtain the updated data in the window, and then the window aggregation calculation corresponding to the task can be performed based on the updated data in the window.

[0080] Furthermore, since the task duration is fixed, that is, the window size is fixed, therefore, for the latest stream data, when it is about to be regarded as expired data as time goes by, in order to specifically explain how to determine the window calculation result triggered when the latest stream data expires, that is, to determine the task output result after the entire task duration has elapsed from the task start time point, step S30 includes steps S301 to S302:

[0081] Step S301: when the current time point meets the time trigger condition of the expiration time point timer of the stream data, a window operation is performed on the data in the window where the stream data is located, and the window calculation result is updated to obtain a window withdrawal calculation result.

[0082] Step S302: determining the window withdrawal calculation result as the task output result.

[0083] It should be understood that as time goes on, until the current time point reaches the expiration time point of the latest stream data, the window size corresponding to a task duration is satisfied. Since the data in the window includes the latest stream data and the non-expired data in the original data of the window, the window aggregation calculation corresponding to the task can be performed based on the data in the window, the existing window calculation results can be updated, the window withdrawal calculation results can be obtained, and the window withdrawal calculation results can be output, that is, the task output result within a complete task duration starting from the task start time point is obtained.

[0084] In addition, this can be combined with Figure 3 Explain the calculation logic of the Flink window in this application. Figure 3 This is a schematic diagram of the Flink window computing logic for this application.

[0085] exist Figure 3 In the Flink window, the Over window can be used to perform aggregate calculations on the data.

[0086] If the task is to count the number of transactions in the last 24 hours, the task duration, that is, the window size, is 24 hours. Taking the time point T when the latest stream data (RowData) arrives at the Flink window as an example, when the stream data arrives, the expiration time point timer corresponding to the stream data is registered at time point T+1ms (that is, it can be regarded as immediate registration when the stream data arrives). Since the number of data in the window changes, Flink SQL immediately performs window aggregation calculation on the data in the window and outputs the window calculation result.

[0087] When registering the expiration time point timer, the event time E, that is, the data generation time (for example, the transaction data source generally contains a transaction time field), is used instead of the processing time T, so that the "expiration time point" is not affected by data delays.

[0088] After a task duration (window size) has passed, that is, when the expiration time point of the stream data is reached (time point E+24h), the window aggregation calculation of the window withdrawal is triggered again, and the window calculation result is also output.

[0089] The above-mentioned time point T can be any time point when different stream data arrives, so that the event time point E corresponding to each stream data can be obtained to de-window and trigger window calculation according to the time interval of time point E + window size. Regardless of the delayed arrival of stream data or the out-of-order arrival of stream data, the final calculation result of Flink can be kept in time sequence consistency, thereby ensuring that the accuracy of Flink calculation reaches the database accuracy level, without the need for Flink WaterMark configuration, and reducing the computing load of Flink jobs.

[0090] In this embodiment, when the window receives the stream data corresponding to the task start time point, the data in the window is updated; based on the update result, the window operation is performed to obtain the window calculation result; when the current time point meets the time trigger condition of the timer of the expiration time point of the original data in the window, the data in the window is updated to withdraw the expired data in the original data in the window; based on the update result, the window operation is performed to obtain the window calculation result; and then when the current time point meets the time trigger condition of the timer of the expiration time point of the stream data, the window operation is performed on the data in the window where the stream data is located, and the window calculation result is updated to obtain the window withdrawal calculation result; the window withdrawal calculation result is determined as the task output result. In this embodiment, the calculation result update can be triggered when the latest data arrives at the window and each data in the window expires, so that the task output result within the complete task duration starting from a task start time point determined based on the latest data can be obtained, which helps to meet the diverse task calculation needs of users.

[0091] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 4 , Figure 4 This is a flow chart of the third embodiment of the stream data processing method based on Flink window of the present application.

[0092] In this embodiment, considering that the calculation result of the Over window is time-sensitive, after step S302, the following steps are further included: steps S401 to S402:

[0093] Step S401: Update the data in the window to withdraw the stream data, which is expired data.

[0094] Step S402: Perform window operation based on the update result to obtain a window calculation result, which is zero.

[0095] It should be understood that when the current time point reaches the expiration time point of the latest stream data and satisfies the window size corresponding to a task duration, since the latest stream data is expired, the stream data that is expired can be removed from the window. At this time, the task-related data in the window has been destroyed, and the obtained window calculation result is zero.

[0096] Furthermore, considering that after Flink outputs the final calculation result, the task-related data in the window has been destroyed and the zeroing operation cannot be triggered, it is difficult to achieve the final calculation result zeroing unless the underlying logic of Flink is greatly changed. Therefore, it is possible to consider combining an external storage system to set the timeliness of the calculation result. After step S302, the following steps are also included:

[0097] Step S501: storing the window calculation result or the task output result to an external system through a Sink operator.

[0098] It should be understood that Flink maintains Source and Sink tables. The Source table is the starting point of the stream data and is responsible for receiving stream data input from the outside. The Sink table is the end point of the data flow and is used to output the processed data to the external system for storage.

[0099] It is understandable that the external system can be, for example, Redis and Hbase. Specifically, in Flink, the calculation results can be output to the external system by using the addSink method. Flink can provide a variety of Sink interfaces, such as RedisSink, HbaseSink, etc., for integration with external systems.

[0100] It should be noted that during the entire task duration, when the number of data in the window changes, the window calculation is triggered, and the window calculation result is obtained. Then, the window calculation result can be stored in the external system in real time through the Sink operator. When the window size corresponding to a task duration is met, the final window withdrawal calculation result, i.e., the task output result, is also stored in the external system.

[0101] It should also be noted that when the window size corresponding to a task duration is met, at this time, since the above-mentioned latest stream data is expired data, the stream data that is expired data can be withdrawn from the window. At this time, the task-related data in the window has been destroyed, and the obtained window calculation result is zero. The window calculation result can also be stored in an external system, and the above-mentioned task output result can be updated to zero.

[0102] Step S502: Determine a zeroing window time point for the task output result based on the TTL service of the external system and the task duration.

[0103] It should be noted that the Time-To-Live (TTL) service is a mechanism that allows data to be stored in the system with a lifetime. When the data reaches this preset time, it will be automatically deleted or marked as expired by the system and will no longer be queried or accessed. The TTL service is a very useful feature for processing time-sensitive data, especially in scenarios where old data needs to be automatically cleaned up to save storage space and maintain data relevance.

[0104] It should be understood that when the final task output result is written to the external system, a TTL value can be specified. This TTL value can usually be consistent with the task duration. For example, if the task duration is to collect data within 24 hours, the TTL can be set to 24 hours.

[0105] Step S503: When the current time point meets the zeroing window time point, trigger a zeroing operation to clear the task output result.

[0106] It is understandable that as time passes, until the current time point reaches the time point corresponding to the TTL set above, the deletion operation can be automatically executed to delete the task output result, and the task output result can be deleted in the external system to save storage space of the external system.

[0107] In this embodiment, a calculation result aging mechanism for Over window aggregation calculation is designed. By combining the TTL services of external systems such as Redis and Hbase, the final variable result zeroing operation is triggered on demand. When the window calculation results or task output results in Flink are stored in the external system through the Sink operator, the result TTL time window is specified to implement the window withdrawal operation on demand.

[0108] In addition, you can Figure 5 Taking the example of the existing method for explaining the window back calculation due to delayed arrival of streaming data, Figure 5 Activate the computational implementation graph for existing data window withdrawal.

[0109] exist Figure 5 In this example, we take Flink's half-hour transaction statistics scenario as an example:

[0110] In normal trading: if transaction 1 enters the window at 9 o'clock, the window calculation is triggered, the window calculation result is updated and output to the external system, so that redis count = 1; from 9 o'clock to 9:30, redis count = 1; until 9:30, transaction 1 exits the window and updates, triggering the exit window calculation, updating the window calculation result and outputting it to the external system, so that redis count = 0; and redis count = 0 between 9:30 and 10 o'clock; until 10 o'clock redis is automatically destroyed, redis = null.

[0111] In delayed trading: the actual transaction time occurs at 9 o'clock, but it is delayed to 9:15 to reach the window. In the interval from 9 o'clock to 9:15, redis count = 0; until 9:15, the actual transaction enters the window, triggering the window calculation, making redis count = 1; from 9:15 to 9:30, redis count = 1; 9:30 is the expected window exit time, but because the data arrives 15 minutes late, the corresponding window exit time is postponed by 15 minutes, and in the interval from 9:30 to 9:45, redis count = 1; until 9:45, the actual window exit time is reached, that is, the actual window exit update is triggered, the window exit calculation is updated and output to the external system, making redis count = 0; and redis count = 0 is maintained between 9:40 and 10:15; until 10:15, redis is automatically destroyed and redis = null.

[0112] In the present application, when registering an expiration time point timer based on the windowed stream data, the event time, i.e., the data generation time, is used instead of the processing time. Specifically, the expiration time point of the stream data is determined in the manner of "expiration time point = event time point + window size", so that the "expiration time point" is not affected by data delays, and the window exit calculation can be triggered on demand at the expected window exit time when the task requirements are met, so as to obtain accurate window calculation results, thereby solving the problem of calculation accuracy under large flow data in the existing method and balancing the resource utilization and calculation accuracy of the stream computing operation.

[0113] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the stream data processing method based on Flink windows of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0114] In addition, the present application also provides a stream data processing device based on Flink window, referring to Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the stream data processing device based on Flink window of this application; Figure 6 As shown, the device comprises:

[0115] The requirement configuration module 601 is used to determine the task starting time point and task duration according to the task requirement information;

[0116] The expiration point registration module 602 is used to obtain the event time point of the stream data when receiving the stream data corresponding to the task start time point, and register the expiration time point timer of the stream data according to the event time point and the task duration;

[0117] The result output module 603 is used to perform a window operation on the data in the window where the stream data is located to obtain a task output result when the time trigger condition of the expiration time point timer is met.

[0118] Furthermore, the expiration point registration module 602 is also used to determine the starting time point of the task, and obtain the event time point of the stream data when the window receives the stream data corresponding to the task time point; determine the expiration time point of the stream data according to the event time point and the task duration; and generate an expiration time point timer for the stream data according to the expiration time point.

[0119] Furthermore, the expiration point registration module 602 is also used to update the data in the window when the window receives the stream data corresponding to the task start time point; perform window operations based on the update results to obtain window calculation results.

[0120] The window includes original data of the window.

[0121] Furthermore, the expiration point registration module 602 is also used to update the data in the window to withdraw the expired data in the original data of the window when the current time point meets the time trigger condition of the expiration time point timer of the original data of the window; perform window operations based on the update results to obtain window calculation results.

[0122] Furthermore, the result output module 603 is also used to perform window operations on the data in the window where the stream data is located when the time trigger condition of the expiration time point timer of the stream data is met at the current time point, and update the window calculation result to obtain a window withdrawal calculation result, wherein the data in the window includes the stream data and non-expired data in the original data of the window; and determine the window withdrawal calculation result as the task output result.

[0123] Furthermore, the result output module 603 is also used to update the data in the window to withdraw the stream data, which is expired data; perform window operation based on the update result to obtain the window calculation result, and the window calculation result is zero.

[0124] Furthermore, the result output module 603 is also used to store the window calculation result or the task output result to an external system through a Sink operator; determine the zeroing window time point of the task output result based on the TTL service of the external system according to the task duration; when the current time point meets the zeroing window time point, trigger the zeroing operation to clear the task output result.

[0125] This embodiment can trigger the Flink window job based on the event time point and the task duration, and can maintain the timing consistency of the Flink window calculation results when the streaming data arrives late, thereby meeting the real-time and high-precision requirements of the task output results.

[0126] Other embodiments or specific implementations of the Flink window-based stream data processing device described in the present application can refer to the above-mentioned method embodiments and will not be described in detail here.

[0127] The present application also provides a Flink window-based stream data processing device, the Flink window-based stream data processing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Flink window-based stream data processing method in the above-mentioned embodiment 1.

[0128] Reference below Figure 7 , Figure 7The structural diagram of the stream data processing device based on Flink window of the present application. The stream data processing device based on Flink window in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The Flink window-based stream data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 7 As shown, the stream data processing device based on the Flink window may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the stream data processing device based on the Flink window are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the Flink window-based stream data processing device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a Flink window-based stream data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or provided alternatively.

[0130] The present application also provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the Flink window-based stream data processing method in the above-mentioned embodiment.

[0131] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0132] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A stream data processing method based on Flink window, characterized in that: The method comprises: Determine the task start time and duration based on task requirement information; When receiving the stream data corresponding to the task start time point, obtaining the event time point of the stream data, and registering the expiration time point timer of the stream data according to the event time point and the task duration; When the time trigger condition of the expiration time point timer is met, a window operation is performed on the data in the window where the stream data is located to obtain a task output result.

2. The method according to claim 1, characterized in that The step of obtaining the event time point of the stream data upon receiving the stream data corresponding to the task start time point, and registering the expiration time point timer of the stream data according to the event time point and the task duration, comprises: Determine the task start time point, and obtain the event time point of the stream data when the window receives the stream data corresponding to the task time point; Determine the expiration time point of the stream data according to the event time point and the task duration; An expiration time point timer for the stream data is generated according to the expiration time point.

3. The method according to claim 2, characterized in that After the step of determining the task start time point, the method further includes: When the window receives the stream data corresponding to the task start time point, the data in the window is updated; Perform window operations based on the update results to obtain window calculation results.

4. The method according to claim 2, characterized in that The window includes original data of the window, and after the step of generating an expiration time point timer of the stream data according to the expiration time point, the step further includes: When the current time point satisfies the time trigger condition of the expiration time point timer of the original data in the window, the data in the window is updated to remove the expired data in the original data in the window; Perform window operations based on the update results to obtain window calculation results.

5. The method according to claim 3 or 4, characterized in that The step of performing a window operation on the data in the window where the stream data is located to obtain a task output result when the time trigger condition of the expiration time point timer is met includes: When the current time point satisfies the time trigger condition of the expiration time point timer of the stream data, a window operation is performed on the data in the window where the stream data is located, and the window calculation result is updated to obtain a window withdrawal calculation result, wherein the data in the window includes the stream data and the non-expired data in the original data of the window; The window withdrawal calculation result is determined as the task output result.

6. The method according to claim 5, characterized in that After the step of determining the window withdrawal calculation result as the task output result, the method further includes: Updating the data in the window to withdraw the stream data, the stream data being expired data; A window operation is performed based on the update result to obtain a window calculation result, where the window calculation result is zero.

7. The method according to claim 5, characterized in that The method further comprises: The window calculation result or the task output result is stored in an external system through a Sink operator; The TTL service based on the external system determines the zeroing window time point of the task output result according to the task duration; When the current time point meets the zeroing window time point, a zeroing operation is triggered to clear the task output result.

8. A stream data processing device based on Flink window, characterized in that: The device comprises: The demand configuration module is used to determine the task start time and task duration according to the task demand information; An expiration point registration module is used to obtain the event time point of the stream data when receiving the stream data corresponding to the task start time point, and register the expiration time point timer of the stream data according to the event time point and the task duration; The result output module is used to perform a window operation on the data in the window where the stream data is located to obtain a task output result when the time trigger condition of the expiration time point timer is met.

9. A stream data processing device based on Flink window, characterized in that: The Flink window-based stream data processing device includes: a memory, a processor, and a Flink window-based stream data processing program stored in the memory and executable on the processor. When the Flink window-based stream data processing program is executed by the processor, the steps of the Flink window-based stream data processing method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a Flink window-based stream data processing program, which, when executed by a processor, implements the steps of the Flink window-based stream data processing method according to any one of claims 1 to 7.

Images