A stream big data platform data real-time processing method and system

Abstract

This invention provides a real-time data processing method and system for a streaming big data platform. The user inputs key time points for the business and a specified delay time. After the streaming big data platform starts, a stack is created. If the Event Time of the current data is less than the Event Time of the previous data, the time difference between the current data's Event Time and the current time is calculated. A structure including the data source, time difference, whether it was discarded, and Event Time is constructed and pushed onto the stack. Every M time windows, the sum of the time differences in the stack is calculated, and all data in the stack is popped. If the next M time windows include key time points and the number of structures is less than a preset value, the delay time is set to 0. Otherwise, the delay time within the next M time windows is obtained based on the sum of the time differences and the specified delay time. This invention not only meets the data integrity requirements in big data processing but also satisfies the real-time requirements.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a method and system for real-time data processing on a streaming big data platform. Background Technology

[0002] With the development of computer technology, an increasing amount of data has emerged, including structured, semi-structured, and unstructured data. This massive amount of data presents significant challenges for data analysis. Whether it's climate change, air pollution, population health risk assessment, or precision medicine, all involve the analysis of large amounts of data. Big data technology has effectively solved this problem. Hadoop, Spark, and Flink are currently the most widely used big data frameworks. Flink is particularly suitable for processing streaming data. Compared to batch data, streaming data is constantly being generated. In some cases, the time order requirements for streaming data are quite strict, such as the real-time collection of weather sensor data or the real-time statistics of hospital visitor numbers. However, due to the varying processing capabilities of different devices and network factors, data generation can easily result in out-of-order data. Out-of-order data refers to data not being sent to the streaming big data platform according to the time of data or event occurrence, which poses a problem for the real-time processing of streaming big data platforms.

[0003] To address out-of-order delivery issues, Flink uses watermarks, which involve several important concepts: Window, EventTime, and ProcessingTime. Window represents the time window, EventTime represents the message occurrence time (a timestamp inherent to the message), and ProcessingTime is the data processing time. Ideally, the data occurrence time equals the data processing time, but this is not always the case in reality. A watermark is a timestamp equal to the difference between the longest arrival time of all arriving data and a specified delay. However, the specified delay is fixed and cannot be adjusted based on business needs, meaning real-time adjustments to data processing are not possible. Summary of the Invention

[0004] To better leverage the real-time performance of streaming big data platforms, this invention provides a real-time data processing method for streaming big data platforms, the method comprising the following steps:

[0005] Step 1: After the user inputs the key time points of the business and the specified delay time, a stack is created after the streaming big data platform is started.

[0006] Step 3: When a piece of data arrives at the streaming big data platform, if the EventTime of the current data is less than the EventTime of the previous data, calculate the time difference between the EventTime of the current data and the current time, construct a structure including the data source, the time difference, whether it is discarded, and the EventTime, and push the structure onto the stack. Calculate the sum of the time differences in the stack every M time windows, and pop all the data from the stack.

[0007] Step 5: If the next M time windows include the key time points and the number of structures is less than a preset value, then the delay time is set to 0; otherwise, the delay time within the next M time windows is obtained based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with EventTime, and M is a positive integer.

[0008] Preferably, the step of obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves:

[0009] The weights of the delay times are determined based on the sum of the time differences.

[0010] If the next M time windows include the key time points and the number of structures is not less than a preset value, then the minimum value of the weight and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time.

[0011] If the next M time windows do not include the key time point, then the weight is used as a weight coefficient, and the product of the weight coefficient and the specified delay time is used as the new delay time.

[0012] Preferably, determining the weight of the delay time based on the sum of the time differences specifically involves:

[0013] Multiple intervals are set, each interval corresponds to a weight, the interval into which the sum of the time differences falls is determined, and the weight is obtained based on the interval into which the sum of the time differences falls.

[0014] Preferably, step 4 is included between step 2 and step 4;

[0015] Step 4: Each time all structures are popped from the stack, the structure is parsed to obtain the data source, the time difference, whether it was discarded, and EventTime. The parsed data is stored as a record in the database of an independent node. When the critical time point is reached, the independent node corrects the results of the streaming big data platform at the previous critical time point based on the data stored in the database.

[0016] Preferably, the independent node corrects the results of the streaming big data platform at the previous key time point based on the data stored in the database, specifically as follows:

[0017] Retrieve the database records of EventTime between the previous key time point and the key time point before that, and obtain the corrected result based on the database records and the result of the previous key time point.

[0018] Preferably, the independent node is also used to count the data sources in the popped data stored in the database and issue a prompt message.

[0019] Furthermore, the present invention also provides a real-time data processing system for a streaming big data platform, the system comprising the following modules:

[0020] The parameter input module is used for users to input key time points of the business and specified delay times. After the streaming big data platform starts, a stack is created.

[0021] The statistics module is used to calculate the time difference between the current data's EventTime and the current time when a piece of data arrives at the streaming big data platform. If the EventTime of the current data is less than the EventTime of the previous data, the module constructs a structure including the data source, the time difference, whether the data was discarded, and the EventTime, and pushes the structure onto a stack. Every M time windows, the module calculates the sum of the time differences in the stack and pops all the data from the stack.

[0022] The delay time calculation module sets the delay time to 0 if the next M time windows include the key time point and the number of structures is less than a preset value; otherwise, it obtains the delay time within the next M time windows based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with EventTime, and M is a positive integer.

[0023] Preferably, the step of obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves:

[0024] The weights of the delay times are determined based on the sum of the time differences.

[0025] If the next M time windows include the key time points and the number of structures is not less than a preset value, then the minimum value of the weight and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time.

[0026] If the next M time windows do not include the key time point, then the weight is used as a weight coefficient, and the product of the weight coefficient and the specified delay time is used as the new delay time.

[0027] Preferably, determining the weight of the delay time based on the sum of the time differences specifically involves:

[0028] Multiple intervals are set, each interval corresponds to a weight, the interval into which the sum of the time differences falls is determined, and the weight is obtained based on the interval into which the sum of the time differences falls.

[0029] Preferably, a correction module is also included between the statistics module and the delay time calculation module;

[0030] The correction module is used to parse the structure each time all structures are popped from the stack, to obtain the data source, the time difference, whether it was discarded, and EventTime, and to store the parsed data as a record in the database of an independent node; when the key time point is reached, the independent node corrects the result of the streaming big data platform at the previous key time point based on the data stored in the database.

[0031] Preferably, the independent node corrects the results of the streaming big data platform at the previous key time point based on the data stored in the database, specifically as follows:

[0032] Retrieve the database records of EventTime between the previous key time point and the key time point before that, and obtain the corrected result based on the database records and the result of the previous key time point.

[0033] Preferably, the independent node is also used to count the data sources in the popped data stored in the database and issue a prompt message.

[0034] Finally, the present invention also provides a computer-readable storage medium on which a computer program is stored, the computer program being executed by a processor to perform the method described above.

[0035] When streaming big data platforms like Flink process streaming data, the order in which data arrives at the platform may be disrupted due to network and other factors. For businesses with high real-time requirements, this will affect the accuracy of the platform's calculation results. Streaming big data platforms use the Watermark mechanism to address this issue. However, this mechanism will delay the calculation at all points in time. Some points in time are critical, and it is desirable to output results as real-time as possible at these critical points. The method provided in this invention can output results in real-time at critical points and as real-time as possible at other points in time, while considering both data integrity and real-time performance, achieving a balance between the two. Attached Figure Description

[0036] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart of one embodiment of the present invention;

[0038] Figure 2 A diagram illustrating how data arrives at the big data platform in sequence;

[0039] Figure 3 A diagram illustrating how disordered data is achieved on a big data platform;

[0040] Figure 4 This is a flowchart of another embodiment of the present invention;

[0041] Figure 5 This is a structural diagram of one embodiment of the present invention. Detailed Implementation

[0042] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention relates to several important concepts in big data platforms, particularly Flink:

[0045] Time window: Used to divide streaming data (stream data, stream) into multiple units over time, with each unit used for one calculation;

[0046] WaterMark: Used for processing out-of-order data. It is calculated based on the set delay time (tolerance delay time) and EventTime, i.e., WaterMark = EventTime - Tolerance Time; its type is timestamp.

[0047] EventTime: The time when the data was generated or the event was generated. It is usually generated at the data source end and is a timestamp.

[0048] ProcessingTime: Data processing time, generally the time it takes for data to be processed by operators in Flink;

[0049] Delay time: also known as tolerance delay time or tolerable delay time, is related to business requirements.

[0050] This invention provides a real-time data processing method for a streaming big data platform, such as... Figure 1 As shown, the method includes the following steps:

[0051] Step 1: After the user inputs the key time points of the business and the specified delay time, a stack is created after the streaming big data platform is started.

[0052] Different business operations have different critical time points. For example, at critical time points for sensors, data needs to be processed as real-time as possible. The specified delay time refers to the general tolerable delay time. A stack is a data structure that operates on a last-in, first-out (LIFO) principle.

[0053] Step 3: When a piece of data arrives at the streaming big data platform, if the EventTime of the current data is less than the EventTime of the previous data, calculate the time difference between the EventTime of the current data and the current time, construct a structure including the data source, the time difference, whether it is discarded, and the EventTime, and push the structure onto the stack. Calculate the sum of the time differences in the stack every M time windows, and pop all the data from the stack.

[0054] Out-of-order data can negatively impact the processing of data by big data platforms. Ideally, the data should be sent in the correct order. Figure 2 As shown, however, in practical applications, the data may become out of order, such as... Figure 3 As shown. Some data arrival times are delayed. For example, data generated in the 1st second (EventTime) may arrive in Flink in the 5th second. The time window is 3 seconds. If no delay time is set, the data cannot be processed within the 0-3 second window. If a delay time of 3 seconds is set, Flink will wait for 3 seconds. If the data has not arrived within these 3 seconds, it is considered that all the data has arrived. This is also a manifestation of data integrity.

[0055] The data itself carries an EventTime, indicating when the data was generated. This EventTime is generated at the data source. Specifically, if a user clicks a button on a mobile device, the EventTime is generated when data is generated, or if a sensor's processor includes the EventTime in the data. If one EventTime occurs before another, it means that the first data was generated earlier and should arrive at the big data platform first.

[0056] The time difference between EventTime and the current time reflects the time from data generation to transmission to the big data platform. It is related to the processing capacity of the source and the network conditions. This invention uses the time difference between EventTime and the current time to reflect the situation of each source.

[0057] In the specific implementation, whether data is discarded is used as a flag to indicate whether the data was processed within its corresponding time window. Some data, although delayed or out of order, may not be discarded within the corresponding time window based on the Watermark and time window criteria, indicating that the data has been processed and participated in the operator calculations of the big data platform. Data with excessively long delays will be discarded within the time window, assuming that all data within the time window has arrived.

[0058] Structures are implemented using the struct method, for example:

[0059]

[0060] Step 5: If the next M time windows include the key time points and the number of structures is less than a preset value, then the delay time is set to 0; otherwise, the delay time within the next M time windows is obtained based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with EventTime, and M is a positive integer.

[0061] At critical time points, real-time requirements are high, while data integrity requirements can be appropriately reduced. Therefore, if the next M time windows include critical time points and the number of structures is less than a preset value, the delay time is set to 0, making the WaterMark the same as the EventTime. The fact that the number of structures is less than the preset value indicates that data integrity is within acceptable limits, thus satisfying both real-time and integrity requirements.

[0062] When significant data delays occur, i.e., when a large amount of data is out of order, it indicates that the overall network is poor. To ensure data integrity, the required delay time should be increased; conversely, to reflect the real-time nature of data processing, the required delay time should be decreased. In a specific embodiment, obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves:

[0063] The weights of the delay times are determined based on the sum of the time differences.

[0064] If the next M time windows include the key time points and the number of structures is not less than a preset value, then the minimum value of the weight and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time.

[0065] If the next M time windows do not include the key time point, then the weight is used as a weight coefficient, and the product of the weight coefficient and the specified delay time is used as the new delay time.

[0066] In one specific embodiment, determining the weight of the delay time based on the sum of the time differences specifically involves:

[0067] Multiple intervals are set, each interval corresponds to a weight, the interval into which the sum of the time differences falls is determined, and the weight is obtained based on the interval into which the sum of the time differences falls.

[0068] Data that was discarded should also be processed to ensure all data is processed. A separate node should be set up to correct the results from the big data platform. This way, the processing performance of the big data platform is not affected, and data correction is achieved. In one specific embodiment, between steps 2 and 4, such as... Figure 4 As shown, it also includes step 4;

[0069] Step 4: Each time all structures are popped from the stack, the structure is parsed to obtain the data source, the time difference, whether it was discarded, and EventTime. The parsed data is stored as a record in the database of an independent node. When the critical time point is reached, the independent node corrects the results of the streaming big data platform at the previous critical time point based on the data stored in the database.

[0070] This invention uses key time points as trigger points for correction processing, which helps business personnel obtain accurate key time point data.

[0071] In one specific embodiment, the independent node corrects the results of the streaming big data platform at the previous key time point based on the data stored in the database, specifically as follows:

[0072] Retrieve the database records of EventTime between the previous key time point and the key time point before that, and obtain the corrected result based on the database records and the result of the previous key time point.

[0073] Different data sources are affected differently by factors such as network connectivity. Some nodes exhibit significant data latency, requiring special attention or investigation of these nodes, such as the network transmitting devices where the sensors reside. In one specific embodiment, the independent node is also used to statistically analyze the data sources in the popped data stored in the database and issue alert messages.

[0074] Furthermore, the present invention also provides a real-time data processing system for a streaming big data platform, the system comprising the following modules:

[0075] The parameter input module is used for users to input key time points of the business and specified delay times. After the streaming big data platform starts, a stack is created.

[0076] The statistics module is used to calculate the time difference between the current data's EventTime and the current time when a piece of data arrives at the streaming big data platform. If the EventTime of the current data is less than the EventTime of the previous data, the module constructs a structure including the data source, the time difference, whether the data was discarded, and the EventTime, and pushes the structure onto a stack. Every M time windows, the module calculates the sum of the time differences in the stack and pops all the data from the stack.

[0077] The delay time calculation module sets the delay time to 0 if the next M time windows include the key time point and the number of structures is less than a preset value; otherwise, it obtains the delay time within the next M time windows based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with EventTime, and M is a positive integer.

[0078] Preferably, the step of obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves:

[0079] The weights of the delay times are determined based on the sum of the time differences.

[0080] If the next M time windows include the key time points and the number of structures is not less than a preset value, then the minimum value of the weight and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time.

[0081] If the next M time windows do not include the key time point, then the weight is used as a weight coefficient, and the product of the weight coefficient and the specified delay time is used as the new delay time.

[0082] Preferably, determining the weight of the delay time based on the sum of the time differences specifically involves:

[0083] Multiple intervals are set, each interval corresponds to a weight, the interval into which the sum of the time differences falls is determined, and the weight is obtained based on the interval into which the sum of the time differences falls.

[0084] Preferably, a correction module is also included between the statistics module and the delay time calculation module, such as... Figure 5 As shown;

[0085] The correction module is used to parse the structure each time all structures are popped from the stack, to obtain the data source, the time difference, whether it was discarded, and EventTime, and to store the parsed data as a record in the database of an independent node; when the key time point is reached, the independent node corrects the result of the streaming big data platform at the previous key time point based on the data stored in the database.

[0086] Preferably, the independent node corrects the results of the streaming big data platform at the previous key time point based on the data stored in the database, specifically as follows:

[0087] Retrieve the database records of EventTime between the previous key time point and the key time point before that, and obtain the corrected result based on the database records and the result of the previous key time point.

[0088] Preferably, the independent node is also used to count the data sources in the popped data stored in the database and issue a prompt message.

[0089] Finally, the present invention also provides a computer-readable storage medium on which a computer program is stored, the computer program being executed by a processor to perform the method described above.

[0090] The present invention also provides an apparatus comprising a computer-readable storage medium and a processor, wherein a computer program is stored on the storage medium, and the computer program is executed by the processor to perform the method described above.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time data processing method for a streaming big data platform, characterized in that, The method includes the following steps: Step 1: After the user inputs the key time points of the business and the specified delay time, a stack is created after the streaming big data platform is started. Step 3: When a piece of data arrives at the streaming big data platform, if the Event Time of the current data is less than the Event Time of the previous data, calculate the time difference between the Event Time of the current data and the current time, construct a structure including the data source, the time difference, whether it is discarded, and the Event Time, and push the structure onto the stack. Calculate the sum of the time differences in the stack every M time windows, and pop all the data from the stack. Step 5: If the next M time windows include the key time points and the number of structures is less than a preset value, then the delay time is set to 0; otherwise, the delay time within the next M time windows is obtained based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with Event Time, and M is a positive integer. The step of obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves: determining the weight of the delay time based on the sum of the time differences; if the next M time windows include the key time point and the number of structures is not less than a preset value, then the minimum value of the sum of the weights and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time; if the next M time windows do not include the key time point, then the weight is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time.

2. The method as described in claim 1, characterized in that, The weighting of the delay time based on the sum of the time differences is specifically as follows: Multiple intervals are set, each interval corresponds to a weight, the interval into which the sum of the time differences falls is determined, and the weight is obtained based on the interval into which the sum of the time differences falls.

3. The method as described in claim 1, characterized in that, Between step 3 and step 5, there is also step 4; Step 4: Each time all structures are popped from the stack, the structure is parsed to obtain the data source, the time difference, whether it was discarded, and the Event Time. The parsed data is stored as a record in the database of an independent node. When the critical time point is reached, the independent node corrects the results of the streaming big data platform at the previous critical time point based on the data stored in the database.

4. The method as described in claim 3, characterized in that, The independent node corrects the results of the streaming big data platform at the previous key time point based on the data stored in the database, specifically as follows: Retrieve the database records of Event Time between the previous key time point and the key time point before that, and obtain the corrected result based on the database records and the result of the previous key time point.

5. The method as described in claim 3, characterized in that, The independent node is also used to count the data source of the popped data stored in the database and issue a prompt message.

6. A real-time data processing system for a streaming big data platform, characterized in that, The system includes the following modules: The parameter input module is used for users to input key time points of the business and specified delay times. After the streaming big data platform starts, a stack is created. The statistics module is used to calculate the time difference between the current data's Event Time and the current time when a piece of data arrives at the streaming big data platform. If the Event Time of the current data is less than the Event Time of the previous data, the module constructs a structure including the data source, the time difference, whether the data was discarded, and the Event Time, and pushes the structure onto a stack. Every M time windows, the module calculates the sum of the time differences in the stack and pops all the data from the stack. The delay time calculation module sets the delay time to 0 if the next M time windows include the key time point and the number of structures is less than a preset value; otherwise, it obtains the delay time within the next M time windows based on the sum of the time differences and the specified delay time; wherein, the delay time is used to calculate the WaterMark with Event Time, and M is a positive integer; The step of obtaining the delay time within the next M time windows based on the sum of the time differences and the specified delay time specifically involves: determining the weight of the delay time based on the sum of the time differences; if the next M time windows include the key time points and the number of structures is not less than a preset value, then the minimum value of the sum of the weights and 0.5 is taken as the weight coefficient, and the product of the weight coefficient and the specified delay time is taken as the new delay time; If the next M time windows do not include the key time point, then the weight is used as a weight coefficient, and the product of the weight coefficient and the specified delay time is used as the new delay time.

7. The system as described in claim 6, characterized in that, A correction module is also included between the statistics module and the delay time calculation module; The correction module is used to parse the structure each time all structures are popped from the stack, to obtain the data source, the time difference, whether it was discarded, and the Event Time. The parsed data is stored as a record in the database of an independent node. When the critical time point is reached, the independent node corrects the results of the streaming big data platform at the previous critical time point based on the data stored in the database.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the method as described in any one of claims 1-5.

Images