Parallel Dynamic Consumption Rate Alignment Method and System in Real-Time Big Data Scenarios

By dynamically adjusting the water level offset Y in real-time big data scenarios, the system instability caused by the time difference between data sources in multi-source data processing is solved, the data consumption rate is aligned, and the system's processing speed and stability are improved.

CN118503288BActive Publication Date: 2026-11-10ECCOM NETWORK SYST CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410569672.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2026-11-10
Estimated Expiration
2044-05-09

AI Technical Summary

Technical Problem

In existing technologies, when processing multi-source data in real-time big data scenarios, the time level difference between data sources cannot be dynamically adjusted, which affects the system's processing speed and stability. In particular, it is difficult to effectively balance the data consumption rate between data sources when data traffic changes.

Method used

By setting dynamic water level alignment parameters, the water level offset Y is dynamically adjusted. Based on the cache size x and the time water level, the maximum allowable time water level difference between data sources is adjusted in real time to ensure data consumption rate alignment between data sources.

Benefits of technology

It improves system stability and fault tolerance, reduces resource consumption, achieves real-time and efficient data processing, and adapts to changes in data sources or computing nodes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118503288B_ABST
    Figure CN118503288B_ABST
Patent Text Reader

Abstract

This invention provides a parallel dynamic consumption rate alignment method and system for real-time big data scenarios, comprising: setting dynamic watermark alignment parameters; obtaining the buffer size x and the current data source operator's time watermark by reporting the time watermark and the amount of data cached by the computation operator through the data source coordinator; selecting the watermark offset Y in the parameter range; receiving the time watermarks of all data source operators, obtaining the maximum allowable time watermark and distributing it to each data source operator; determining whether the maximum allowable time watermark is greater than the current data source operator's time watermark; if yes, ingesting data and restarting the determination every time its own time watermark changes; if no, stopping; re-executing steps S2 to S5 until alignment is complete. This invention improves the system's fault tolerance and stability by dynamically adjusting the alignment strategy parameters by obtaining the size of the system's cached data in real time, and the system automatically adjusting the parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a method and system for aligning parallel dynamic consumption rates in real-time big data scenarios. Specifically, it is a method for ensuring consistent data consumption rates to maintain stable task operation when multiple data streams are read simultaneously in a real-time big data stream processing scenario. Background Technology

[0002] With the advent of the big data era, organizations and enterprises face the challenge of processing massive amounts of data. Data alignment during dynamic computation has become a crucial step in achieving efficient data processing and analysis. Many application scenarios require computation and analysis of real-time data, such as financial transactions, the Internet of Things (IoT), and online advertising. Due to the sheer volume and diverse sources of data, data in the same format often needs to be integrated from multiple data sources for aggregation and calculation during computation and analysis. In real-time scenarios, consumption alignment is necessary before unified analysis can be performed. Dynamic data consumption alignment during computation ensures the accuracy and consistency of real-time data during the computation process.

[0003] In real-time, multi-source big data computing scenarios, if data skew or uneven data production speeds occur at the data source layer, some data sources may ingest data at events much faster than others. For statistical computing based on event time, this data with earlier events needs to be cached to wait for data from slower-producing data sources. However, an excessive amount of cached data can slow down the system's data processing speed.

[0004] In summary, a method is needed to dynamically adjust the allowable water level offset based on the size of the cached data in the downstream computing operator. When the data volume of the data source is small or the time difference between events between data sources is small, the system can fully utilize its data processing performance. When an imbalance occurs between data sources, resulting in excessive cached data, the maximum allowable offset can be reduced to allow the data sources to quickly restore balance.

[0005] Currently, a time watermark alignment method exists for streaming data processing in real-time big data scenarios to handle uneven data reading rates from multiple sources. The specific implementation is as follows: taking two data sources as an example, an allowable watermark offset Y is defined for each data source. The minimum time watermark value of the two data sources is periodically obtained, and then added to the minimum allowable watermark offset Y before being pushed to each data source. If the time watermark value of a data source exceeds this value, data ingestion from that data source is stopped.

[0006] This approach limits excessive discrepancies in time watermarks between data sources. However, the volume of data processed by the system varies throughout the day. During certain periods, data volume is high. In these situations, if the maximum allowable offset is set too large, it fails to reduce the cache size of the computational operators. Conversely, if the maximum allowable offset is reduced, during periods of low data volume, alignment constraints frequently cause the system to stop ingesting data, hindering system performance, impacting data processing speed and stability, and reducing work efficiency. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for aligning parallel dynamic consumption rates in real-time big data scenarios.

[0008] A parallel dynamic consumption rate alignment method in a real-time big data scenario provided by the present invention includes:

[0009] Step S1: Set the dynamic water level alignment parameters to obtain the parameter range;

[0010] Step S2: By reporting the time watermark and the amount of data cached by the calculation operator through the data source coordinator, the cache size x and the time watermark of the current data source operator are obtained;

[0011] Step S3: Based on the cache size x, select the water level offset Y in the parameter range;

[0012] Step S4: Receive the time water level lines of all data source operators through the data source coordinator, and based on the water level line offset Y, determine the maximum allowable time water level line and send it to each data source operator.

[0013] Step S5: Determine whether the maximum allowable time water level is greater than the time water level of the current data source operator; if yes, then ingest data and restart the determination every time the time water level changes; if no, then stop.

[0014] Step S6: Repeat steps S2 to S5 until alignment is complete.

[0015] Preferably, in step S1:

[0016] Setting dynamic water level alignment parameters specifically refers to setting the correspondence between the task cache size x and the water level offset Y according to task needs, and defining multiple levels, i.e. multiple parameter ranges.

[0017] Preferably, in step S2: the time watermark of the current data source operator is reported by the data source coordinator, and the time watermark is reported periodically during subsequent transmission. In addition, the amount of cached data is reported to the data source coordinator by calculating the operator, and the cache size x is reported periodically during subsequent data reading.

[0018] Preferably, in step S3: based on the cache size x, the big data processing system selects the water level offset Y as the dynamic water level alignment parameter for the task according to the corresponding parameter range.

[0019] Preferably, in step S4:

[0020] The maximum permissible time water level is the sum of the minimum time water level among all the time water level lines of the data source operators obtained in step S4 and the water level line offset Y.

[0021] A parallel dynamic consumption rate alignment system for real-time big data scenarios, provided by the present invention, includes:

[0022] Module M1: Sets dynamic water level alignment parameters to obtain the parameter range;

[0023] Module M2: By reporting the time watermark and the amount of data cached by the calculation operator through the data source coordinator, the cache size x and the time watermark of the current data source operator are obtained;

[0024] Module M3: Based on the cache size x, select the water level offset Y in the parameter range;

[0025] Module M4: Receives the time water level lines of all data source operators through the data source coordinator, and calculates the maximum allowable time water level line based on the water level line offset Y.

[0026] Module M5: Determines whether the maximum allowable time water level is greater than the time water level of the current data source operator; if yes, it ingests data and restarts the judgment every time its own time water level changes; if no, it stops.

[0027] Module M6: Re-execute modules M2 through M5 until alignment is complete.

[0028] Preferably, in module M1:

[0029] Setting dynamic water level alignment parameters specifically refers to setting the correspondence between the task cache size x and the water level offset Y according to task needs, and defining multiple levels, i.e. multiple parameter ranges.

[0030] Preferably, in module M2: the time watermark of the current data source operator is reported by the data source coordinator, and the time watermark is reported periodically during subsequent transmission. In addition, the amount of cached data is reported to the data source coordinator by calculating the operator, and the cache size x is reported periodically during subsequent data reading.

[0031] Preferably, in module M3: based on the cache size x, the big data processing system selects the water level offset Y as the dynamic water level alignment parameter for the task according to the corresponding parameter range.

[0032] Preferably, in module M4:

[0033] The maximum permissible time water level is the sum of the minimum time water level among all the time water level lines of the data source operators obtained by module M4 and the water level offset Y.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. This invention dynamically adjusts alignment strategy parameters by acquiring the size of the system's cached data in real time. The system automatically adjusts the parameters, reducing the need for manual intervention and enabling it to better cope with failures or changes in data sources or computing nodes, thereby improving the system's fault tolerance and stability.

[0036] 2. This invention improves data processing speed by reducing the amount of cached data.

[0037] 3. This invention enables real-time calculation and analysis by aligning data in a timely manner during the calculation process, thus meeting the requirements of scenarios with real-time data requirements.

[0038] 4. This invention effectively utilizes computing resources and avoids unnecessary data redundancy and resource consumption by adjusting cache and resource overhead in real time. Attached Figure Description

[0039] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0040] Figure 1 This is a schematic diagram of the process provided by the present invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] This invention dynamically adjusts the maximum allowable gap between data source water levels by acquiring the data source water level and the amount of data cached by the downstream operator, thereby aligning the data consumption progress between data sources. The following describes the process and working principle of this solution:

[0043] In the process of proposing this invention, three main technical challenges were addressed;

[0044] Specifically, in the existing scheme, the value of the water level offset is single and fixed, which makes it impossible to implement a dynamic adjustment strategy and reduces the stability of the data processing system.

[0045] To address the first technical challenge, this invention establishes multiple levels, or intervals, by setting a correspondence between the task cache size x and the water level offset Y, allowing the big data processing system to dynamically adjust the water level offset Y, which serves as an alignment strategy parameter. However, this leads to a second technical challenge: dynamic adjustment requires real-time acquisition of the cache size x and the current time water level to adjust the alignment strategy parameter accordingly; otherwise, dynamic alignment is impossible.

[0046] Addressing the second technical challenge, this invention uses a computational operator to report the amount of cached data to the data source coordinator. During subsequent data reading, the cache size *x* is periodically reported. The big data processing system reports the current data source operator's time watermark to the data source coordinator, and periodically reports the time watermark during subsequent transmissions. Based on the cache size *x* and alignment parameters, the data source coordinator selects the watermark offset *Y*, thus solving the dynamic alignment problem. However, this leads to the third technical challenge: the data source operator needs to dynamically adjust its data ingestion rate according to the data source coordinator.

[0047] Addressing the third technical challenge, this invention, through a data source coordinator, receives the time watermarks from all data source operators. Based on the watermark offset Y, it calculates the maximum permissible time watermark and distributes it to each data source operator. The maximum permissible time watermark is then compared with the current operator's time watermark to determine if the maximum permissible time watermark is larger. If so, data intake continues. Each time the time watermark changes, it is again determined whether data intake should be stopped; otherwise, data intake is stopped. This method adjusts the data intake rate of the data source operators in a timely manner, improving work efficiency and system stability.

[0048] This solution specifically reduces the water level offset Y in the following way;

[0049] Before the task starts, several pairs of relationships are established between the cache size (x) and the waterline offset (Y). Specifically, this defines the appropriate waterline offset (Y) for different cache sizes (x) within a certain range. For example, if the cache size (x) is between zero and a certain value, the waterline offset (Y) is set relatively large (e.g., 10 seconds). If the cache size (x) is a large amount that the system cannot handle (e.g., 1GB), the waterline offset (Y) is set as small as possible (e.g., 1ms). Multiple different levels are also set for intermediate ranges. During the data processing task, the cache size (x) is periodically checked, and then the corresponding waterline offset (Y) is selected from the settings and set at the data source.

[0050] A parallel dynamic consumption rate alignment method in a real-time big data scenario provided by the present invention includes:

[0051] Step S1: Set dynamic water level alignment parameters

[0052] Specifically, before starting a data processing task, the user sets the correspondence between the task cache data size x and the water level offset Y. For example, when the cache size x is less than c1, the water level offset Y is selected as d1; when the cache size x is greater than c1 and less than c2, the water level offset Y, i.e. the water level alignment parameter, is selected as d2.

[0053] Step S2: Data source time water level line reporting;

[0054] After the data processing task begins execution through the big data processing system, the data source operator of the task pulls data from the external system and reports the current time water level of the operator to the data source coordinator. During the subsequent sending process, it periodically reports its own time water level to the data source coordinator.

[0055] In other words, the data sources, such as data source 1, data source 2, data source 3, etc., start sending data, and the operators of each data source periodically report their current time watermarks to the data source coordinator; for example, data source 1 is a, data source 2 is b, and data source 3 is c.

[0056] Step S3: Calculate and report the amount of operator data cached;

[0057] Specifically, the computational operators in the data processing tasks developed by the user read data from various data sources, process business data, and periodically report the size of the data cached by itself, i.e., the cache size x.

[0058] Specifically, the process of processing business data involves calculating the business data according to the logic of the task developer and obtaining the result.

[0059] The computation operator reads data from all data sources and performs corresponding data caching based on actual statistical needs, so that all relevant data can be aggregated and calculated under the same scenario. At this time, the computation operator reports the amount of cached data to the data source coordinator, and continues to periodically report the size of the cached data during subsequent data reading processes.

[0060] Step S4: Based on the dynamic water level alignment parameter settings, select the water level offset Y through the data source coordinator;

[0061] Specifically, after the data source coordinator receives the cached data size of the computation operator, i.e., the cache size x, the data processing system selects the corresponding water level offset Y in the parameter range, i.e., the appropriate allowable water level offset Y, according to the water level alignment parameter settings configured by the user for the task in step S1.

[0062] After receiving the cache size x of the computation operator, the data source coordinator selects an appropriate allowable water level offset Y based on the user's settings.

[0063] Specifically, the relationship between cache size x and waterline offset Y is as follows: when the waterline offset Y is large, the cache size x becomes large, requiring the big data processing system to reduce the waterline offset Y to decrease the cache size x. However, when the waterline offset Y is small, the data acquisition speed of the big data processing system slows down. Therefore, it should be allowed for the big data processing system to automatically increase the waterline offset Y when the cache size x is small.

[0064] This feedback mechanism ensures that the big data processing system operates stably when the data volume per second is large (i.e., when the cache size x is greater than or equal to 1GB), and processes data faster when the data volume per second is small (i.e., when the cache size x is less than 1GB). In other words, a cache size of 1GB or greater constitutes a large cache.

[0065] Specifically, after the big data processing system receives the cache size x of the computation operator, it finds which interval the cache size x belongs to in step S1, obtains the corresponding offset parameter, and updates the value of the water level offset Y.

[0066] Step S5: The data source coordinator sends the water level offset Y;

[0067] After receiving the time water level lines from all data source operators, the data source coordinator calculates and obtains time water level line a, time water level line b, and time water level line c, takes the minimum value among them, and adds the allowable water level line offset Y to obtain the maximum allowable time water level line, i.e., min(a,b,c)+Y, and sends it to each data source operator.

[0068] In other words, after receiving the time watermarks of all data source operators, the data source coordinator calculates the minimum value among them and adds the allowed watermark offset Y as the maximum allowed time watermark to be sent to each data source operator.

[0069] Step S5: The data source dynamically adjusts and sends the data;

[0070] Specifically, the data source operator receives the maximum allowed time watermark sent by the data source coordinator, compares the maximum allowed time watermark with the current operator's time watermark, and if the value of the maximum allowed time watermark is larger, it continues to ingest data. Each time its own time watermark changes, it again determines whether to stop; otherwise, it stops. It then waits to execute step S2.

[0071] In other words, after data source operators a, b, and c receive the maximum permissible time watermark sent by the data source coordinator, the value of the maximum permissible time watermark is min(a, b, c) + Y. This value is compared with the current operator's own time watermark. If the maximum permissible time watermark is greater than the current operator's own time watermark, then data intake continues, and a check is performed again each time the current operator's own time watermark changes to determine whether to stop; otherwise, the process stops. Waiting to execute step S2.

[0072] The present invention also provides a parallel dynamic consumption rate alignment system in a real-time big data scenario. The parallel dynamic consumption rate alignment system in a real-time big data scenario can be implemented by executing the process steps of the parallel dynamic consumption rate alignment method in a real-time big data scenario. That is, those skilled in the art can understand the parallel dynamic consumption rate alignment method in a real-time big data scenario as a preferred embodiment of the parallel dynamic consumption rate alignment system in a real-time big data scenario.

[0073] A parallel dynamic consumption rate alignment system for real-time big data scenarios, provided by the present invention, includes:

[0074] Module M1: Sets dynamic water level alignment parameters to obtain the parameter range;

[0075] Module M2: By reporting the time watermark and the amount of data cached by the calculation operator through the data source coordinator, the cache size x and the time watermark of the current data source operator are obtained;

[0076] Module M3: Based on the cache size x, select the water level offset Y in the parameter range;

[0077] Module M4: Receives the time water level lines of all data source operators through the data source coordinator, and calculates the maximum allowable time water level line based on the water level line offset Y.

[0078] Module M5: Determines whether the maximum allowable time water level is greater than the time water level of the current data source operator; if yes, it ingests data and restarts the judgment every time its own time water level changes; if no, it stops.

[0079] Module M6: Re-execute modules M2 through M5 until alignment is complete.

[0080] Specifically, in module M1:

[0081] Setting dynamic water level alignment parameters specifically refers to setting the correspondence between the task cache size x and the water level offset Y according to task needs, and defining multiple levels, i.e. multiple parameter ranges.

[0082] Specifically, in module M2: the time watermark of the current data source operator is reported by the data source coordinator, and the time watermark is reported periodically during subsequent transmission. In addition, the amount of cached data is reported to the data source coordinator by calculating the operator, and the cache size x is reported periodically during subsequent data reading.

[0083] Specifically, in module M3: based on the cache size x, the big data processing system selects the water level offset Y as the dynamic water level alignment parameter for the task according to the corresponding parameter range.

[0084] Specifically, in module M4:

[0085] The maximum permissible time water level is the sum of the minimum time water level among all the time water level lines of the data source operators obtained by module M4 and the water level offset Y.

[0086] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0087] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for aligning parallel dynamic consumption rates in real-time big data scenarios, characterized in that, include: Step S1: Set the dynamic water level alignment parameters to obtain the parameter range; Step S2: By reporting the time watermark and the amount of data cached by the calculation operator through the data source coordinator, the cache size x and the time watermark of the current data source operator are obtained; Step S3: Based on the cache size x, select the water level offset Y in the parameter range; Step S4: Receive the time water level lines of all data source operators through the data source coordinator, and based on the water level line offset Y, determine the maximum allowable time water level line and send it to each data source operator. Step S5: Determine whether the maximum allowable time water level is greater than the time water level of the current data source operator; If yes, then data is collected, and the judgment is restarted each time the water level changes over its own time; otherwise, it stops. Step S6: Repeat steps S2 to S5 until alignment is complete; In step S1: setting dynamic water level alignment parameters, specifically, according to task requirements, setting the correspondence between task buffer size x and water level offset Y, and defining multiple levels, i.e. multiple parameter ranges.

2. The parallel dynamic consumption rate alignment method in real-time big data scenarios according to claim 1, characterized in that, In step S2: the time watermark of the current data source operator is reported by the data source coordinator, and the time watermark is reported periodically during subsequent transmission. In addition, the amount of cached data is reported to the data source coordinator by calculating the operator, and the cache size x is reported periodically during subsequent data reading.

3. The parallel dynamic consumption rate alignment method in real-time big data scenarios according to claim 2, characterized in that, In step S3: Based on the cache size x, the big data processing system selects the water level offset Y as the dynamic water level alignment parameter for the task according to the corresponding parameter range.

4. The parallel dynamic consumption rate alignment method in real-time big data scenarios according to claim 3, characterized in that, In step S4: the maximum allowable time water level is the sum of the minimum time water level among all the time water level lines of the data source operators obtained in step S4 and the water level line offset Y.

5. A parallel dynamic consumption rate alignment system for real-time big data scenarios, characterized in that, include: Module M1: Sets the dynamic water level alignment parameters to obtain the parameter range; Module M2: By reporting the time watermark and the amount of data cached by the calculation operator through the data source coordinator, the cache size x and the time watermark of the current data source operator are obtained; Module M3: Based on the cache size x, select the water level offset Y in the parameter range; Module M4: Receives the time water level lines of all data source operators through the data source coordinator, and calculates the maximum allowable time water level line based on the water level line offset Y. Module M5: Determines whether the maximum allowable time water level is greater than the time water level of the current data source operator; If yes, then data is collected, and the judgment is restarted each time the water level changes over its own time; otherwise, it stops. Module M6: Re-execute modules M2 through M5 until alignment is complete; In module M1: set dynamic water level alignment parameters, specifically, according to task requirements, set the correspondence between task buffer size x and water level offset Y, and define multiple levels, i.e. multiple parameter ranges.

6. The parallel dynamic consumption rate alignment system in a real-time big data scenario according to claim 5, characterized in that, In module M2: the time watermark of the current data source operator is reported by the data source coordinator, and the time watermark is reported periodically during subsequent transmission. In addition, the amount of cached data is reported to the data source coordinator by calculating the operator, and the cache size x is reported periodically during subsequent data reading.

7. The parallel dynamic consumption rate alignment system in a real-time big data scenario according to claim 6, characterized in that, In module M3: based on the cache size x, the big data processing system selects the water level offset Y as the dynamic water level alignment parameter for the task according to the corresponding parameter range.

8. The parallel dynamic consumption rate alignment system in a real-time big data scenario according to claim 7, characterized in that, In module M4: the maximum allowable time water level is the sum of the minimum time water level among all the time water level lines of the data source operators obtained by module M4 and the water level line offset Y.

Citation Information

Patent Citations

  • Real-time data acquisition processing and dump system based on flink

    CN116010454A

  • Cache updating method and related device

    CN117573388A