A streaming data processing method and system

Abstract

This invention relates to a streaming data processing method that performs real-time processing of streaming data. It distributes the data to various categories of cached groups based on preset grouping conditions, achieving aggregation and compression. An innovative data temperature detection design is introduced to obtain the real-time data temperature corresponding to each category of cached groups. Based on the real-time data temperature and a preset lower temperature threshold, the method dynamically adjusts the window size of the data space occupied by each category of cached groups to further improve the streaming data aggregation and compression ratio. This increases the compression rate while reducing resource utilization. Furthermore, the system is designed using a modular approach to improve data processing throughput without increasing hardware resources, thus meeting business requirements.

Description

Technical Field

[0001] This invention relates to a streaming data processing method and system, belonging to the field of data caching processing technology. Background Technology

[0002] Many enterprises and organizations in the industry use window computing to aggregate and compress streaming data. In this business scenario, as the data traffic and time window increase, the amount of computing and resource consumption will increase significantly. When the data traffic or time window size reaches a certain level, it may not even be able to meet the performance of the business system and the data aggregation effect.

[0003] Currently, widely used window aggregation compression calculation solutions in the industry include rolling windows. This involves receiving data, selecting window partition time fields (processing time, data time), dividing the data into multiple groups according to grouping conditions, distributing the data to the corresponding windows for caching and aggregation calculations, and triggering the window when the time of each cache partition reaches the set trigger time, outputting the result data. Streaming data continuously flows into each window for aggregation calculations, repeating the above calculation steps to complete the window aggregation compression calculation of streaming data. This solution mechanically segments streaming data according to time, destroying the inherent heat characteristics of the generated data. With increased data traffic and window size, device resource consumption rises sharply, and the improvement in aggregation compression effect is not significant, failing to meet business needs.

[0004] Another approach involves session windows. For incoming data, a session time field (processing time, data time) is selected, and a session is created based on the grouping field. Data is then distributed to the corresponding windows for caching and aggregation calculations according to the grouping conditions. Once the data in each cached partition reaches the set trigger time (session ends, no new session operations are generated), the window is triggered, and the result data is output. Streaming data continuously flows into each window for aggregation calculations, repeating the above calculation steps to complete the window aggregation and compression calculation of streaming data. This approach offers a significant improvement over rolling windows. As long as new data is generated within the set time period after creating a session for the corresponding group, the window can continuously perform aggregation and compression operations. To a certain extent, it leverages the inherent hot nature of the data to improve the aggregation and compression effect. However, the time boundary is still a one-size-fits-all approach, ignoring scenarios where the next session might immediately generate data, and therefore cannot adequately meet business needs in specific scenarios. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a streaming data processing method that, based on data grouping and aggregation compression, introduces the concept of data temperature to characterize the activity of data groups, further improves the data aggregation compression ratio, increases the throughput of data processing, and meets business requirements.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a streaming data processing method, which performs the following steps A to D in real time for streaming data generated in real time;

[0007] Step A. Distribute the newly generated data objects to the corresponding category cache groups according to the preset grouping conditions, and then proceed to Step B;

[0008] Step B. Obtain the data temperature corresponding to the current moment for each category of cache group, and then proceed to step C;

[0009] Step C. For each category cache group, firstly, based on having the same preset characteristics, aggregate each data object in the category cache group, then compress the data of each data object in the category cache group, update each data object in the category cache group, then update each data object in each category cache group, and then proceed to step D.

[0010] Step D. For each category of cache group, determine whether the data temperature corresponding to the category cache group at the current time is lower than the preset lower limit threshold. If so, release each data object in the category cache group; otherwise, do not process the category cache group.

[0011] As a preferred technical solution of the present invention: In step A, based on the HASH value of the device ID, data objects generated by the same device are distributed to the same category cache group through hashing, which is a preset grouping condition, and each newly generated data object is distributed to the corresponding category cache group.

[0012] As a preferred technical solution of the present invention: In step B, for each category of cache group, if the category of cache group is newly generated at the current time, the initial data temperature corresponding to the category of cache group is calculated based on each data object in the category of cache group, and is used as the data temperature of the category of cache group at the current time; if the category of cache group is not newly generated at the current time, the data temperature of the category of cache group at the current time is calculated based on the data temperature of the category of cache group at the previous time, and based on each data object in the category of cache group; after obtaining the data temperature of each category of cache group at the current time, proceed to step D.

[0013] As a preferred embodiment of the present invention: In step B, if the category cache group is newly generated at the current time, then according to each data object in the category cache group, the following formula is used:

[0014] T0=2α / (1+e -α*count )

[0015] The initial data temperature T0 corresponding to the cache group of this category is calculated and used as the data temperature of the cache group of this category at the current time; where α represents the preset temperature rise and fall coefficient, e represents the natural constant, and count represents the number of data objects distributed to the cache group of this category.

[0016] As a preferred embodiment of the present invention: In step B, if the category cache group is not newly generated at the current time, then the data temperature T corresponding to the previous time n-1 of the category cache group is used. n-1 Based on the data objects in the cache group for that category, the following formula is used:

[0017] T n =T n-1 (1-α)+2α / (1+e -α*count )

[0018] Calculate the data temperature T corresponding to the current time n for this category of cached group. n , where α represents the preset temperature rise / fall coefficient, e represents the natural constant, and count represents the number of data objects distributed to the cache group of this category.

[0019] Corresponding to the above, the technical problem that this invention also needs to solve is to provide a system for a streaming data processing method, which applies a modular design concept, executes the designed data processing method, introduces the concept of data temperature to characterize the activity of data grouping, further improves the data aggregation and compression ratio, improves the throughput of data processing, and meets business requirements.

[0020] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a system for a streaming data processing method, including a data access module, a data aggregation module, a data compression module, a temperature measurement module, a window adjustment module, and a result output module;

[0021] The data access module is used to execute step A, which enables the data object to be sent to the corresponding category cache group;

[0022] The temperature measurement module is used to execute step B to obtain the data temperature of each category of cache group at the current time.

[0023] The data aggregation module is used to perform step C, which aggregates each data object in each category of cache group.

[0024] The data compression module is used to compress each data object in each category of cache group in step C.

[0025] The window adjustment module controls the window size based on the data space occupied by each type of cache group and the handling of whether or not each data object in each type of cache group is released in step D.

[0026] The output module releases each data object in the category cache group that is below the preset lower temperature threshold, according to the steps used to execute step D.

[0027] The streaming data processing method and system described in this invention, compared with the prior art, have the following technical advantages:

[0028] This invention presents a streaming data processing method that performs real-time processing of streaming data. It distributes data to various categories of cached groups based on preset grouping conditions, achieving aggregation and compression. An innovative data temperature detection design is introduced to obtain the real-time data temperature corresponding to each category of cached groups. Based on the real-time data temperature and a preset lower temperature threshold, the method dynamically adjusts the window size of the data space occupied by each category of cached groups to determine whether to release each data object. This further improves the aggregation and compression ratio of streaming data, increasing compression efficiency while reducing resource utilization. Furthermore, the modular design of the system enhances data processing throughput without increasing hardware resources, meeting business requirements. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the streaming data processing method of the present invention;

[0030] Figure 2 This is a schematic diagram of the system architecture of the streaming data processing method of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0032] This invention presents a streaming data processing method for real-time generated streaming data, such as... Figure 1 As shown, steps A through D are executed in real time.

[0033] Step A. Distribute the newly generated data objects to the corresponding category cache groups according to the preset grouping conditions, and then proceed to Step B.

[0034] The preset grouping conditions here, such as designing a hash value based on the device ID, distribute data objects generated by the same device to the same category cache group through hashing, and use this as a preset grouping condition to distribute each newly generated data object to the corresponding category cache group.

[0035] Step B. Obtain the data temperature corresponding to each category of cache group at the current moment. In practical applications, the following steps are performed for each category of cache group:

[0036] If the category cache group is newly generated at the current time, then based on each data object in the category cache group, the following formula applies:

[0037] T0=2α / (1+e -α*count )

[0038] The initial data temperature T0 corresponding to the cache group of this category is calculated and used as the data temperature of the cache group of this category at the current time; where α represents the preset temperature rise and fall coefficient, which is calculated and evaluated through data characteristics over a period of time, and is generally in the range of 0.05 to 0.2; e represents the natural constant, which is approximately equal to 2.718; count represents the number of data objects distributed to the cache group of this category.

[0039] If the category cache group is not newly generated at the current time, then the data temperature T corresponding to the previous time n-1 is used. n-1 Based on the data objects in the cache group for that category, the following formula is used:

[0040] T n =T n-1 (1-α)+2α / (1+e -α*count )

[0041] Calculate the data temperature T corresponding to the current time n for this category of cached group. n .

[0042] After obtaining the data temperature corresponding to the current moment for each category of cache group, proceed to step C.

[0043] Step C. For each category cache group, firstly, based on having the same preset characteristics, aggregate the data objects in the category cache group, then compress the data of each data object in the category cache group, update each data object in the category cache group, and then update each data object in each category cache group, and proceed to step D.

[0044] Regarding aggregation operations, we perform aggregation calculations on data objects with the same characteristics within a category cache group. The specific aggregation logic is customized according to the business requirements. For example, for student information-related calculations, grouping by class information, calculating the count of male and female students in a category cache group, etc., these logics can all be implemented by customizing the aggregation function.

[0045] In practical applications, data compression involves distributing data objects in the same category of cache group to the same compression operator node. Within the compression operator, data compression calculations are performed on each data object, preserving the compression information. For example, for each data object in the category cache group, a count calculation is performed, and only one record of data with the same key needs to be retained, thus achieving the effect of data compression.

[0046] Step D. For each category of cache group, determine whether the data temperature corresponding to the category cache group at the current time is lower than the preset lower limit threshold. If so, release each data object in the category cache group; otherwise, do not process the category cache group.

[0047] In practical applications, the streaming data processing method designed in the above technical solution involves a window formed by the data space occupied by each category of cached groups. During the real-time execution of steps A to D, the window retains a portion of the category cached groups with higher temperatures, waiting for the next moment. If streaming data no longer enters the window, the corresponding category cached groups cool down. Furthermore, some category cached groups persist within the window, and temperature iteration is performed based on the hot and cold characteristics of the category cached groups to achieve dynamic adjustment of the window size. Category cached groups that continuously heat up can dynamically increase the window size, continuously aggregating and compressing within the window. Category cached groups that continuously cool down are output from the data window when they reach a low-temperature threshold, releasing memory resources in a timely manner. In addition, when the window construction conditions (such as the time range) change, triggering the construction of a new window, data with temperatures higher than the threshold are extended to the window at the next moment, expanding the window to the next moment. Category cached groups with temperatures lower than the threshold trigger window output.

[0048] Regarding the streaming data processing method involved in the above technical solution, a system for executing this method was further designed, such as... Figure 2 As shown, it specifically includes a data access module, a data aggregation module, a data compression module, a temperature measurement module, a window adjustment module, and a result output module.

[0049] The data access module is used to execute step A, which enables the data object to be sent to the corresponding category cache group;

[0050] The temperature measurement module is used to execute step B to obtain the data temperature of each category of cache group at the current time.

[0051] The data aggregation module is used to perform step C, which aggregates each data object in each category of cache group.

[0052] The data compression module is used to compress each data object in each category of cache group in step C.

[0053] The window adjustment module controls the window size based on the data space occupied by each type of cache group and the handling of whether or not each data object in each type of cache group is released in step D.

[0054] The output module releases each data object in the category cache group that is below the preset lower temperature threshold, according to the steps used to execute step D.

[0055] The aforementioned technical solution designs a streaming data processing method that performs real-time processing of streaming data. It distributes data to various categories of cache groups based on preset grouping conditions, achieving aggregation and compression respectively. Furthermore, it introduces an innovative data temperature detection design to obtain the real-time data temperature corresponding to each category of cache group. Based on the real-time data temperature and a preset lower temperature threshold, it personalizes the handling of whether to release each data object in each category of cache group, dynamically adjusting the window size of the data space occupied by each category of cache group. This further improves the streaming data aggregation and compression ratio, increasing the compression rate while reducing resource utilization. Moreover, it applies modular design principles to improve the data processing throughput without increasing hardware resources, meeting business requirements.

[0056] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A streaming data processing method, characterized in that, For the real-time generated streaming data, execute steps A through D in real time; Step A. Distribute the newly generated data objects to the corresponding category cache groups according to the preset grouping conditions, and then proceed to Step B; Step B. For each category of cache group, if the category of cache group is newly generated at the current time, calculate the initial data temperature corresponding to the category of cache group based on each data object in the category of cache group, and use it as the data temperature of the category of cache group at the current time. If the category cache group is not newly generated at the current time, then based on the data temperature of the category cache group at the previous time, the data temperature of the category cache group at the current time is calculated according to each data object in the category cache group; After obtaining the data temperature corresponding to the current moment for each category of cache group, proceed to step C; Step C. For each category cache group, firstly, based on having the same preset characteristics, aggregate each data object in the category cache group, then compress the data of each data object in the category cache group, update each data object in the category cache group, then update each data object in each category cache group, and then proceed to step D. Step D. For each category of cache group, determine whether the data temperature corresponding to the category cache group at the current time is lower than the preset lower limit threshold. If so, release each data object in the category cache group; otherwise, do not process the category cache group.

2. The streaming data processing method according to claim 1, characterized in that: In step A, based on the HASH value of the device ID, data objects generated by the same device are distributed to the same category cache group through hashing, which is a preset grouping condition. The newly generated data objects are then distributed to the corresponding category cache groups.

3. The streaming data processing method according to claim 1, characterized in that: In step B, if the category cache group is newly generated at the current time, then based on each data object in the category cache group, the following formula is used: T0=2α / (1+e -α*count ) The initial data temperature T0 corresponding to the cache group of this category is calculated and used as the data temperature of the cache group of this category at the current time; where α represents the preset temperature rise and fall coefficient, e represents the natural constant, and count represents the number of data objects distributed to the cache group of this category.

4. The streaming data processing method according to claim 1, characterized in that: In step B, if the category cache group is not newly generated at the current time, then the data temperature T corresponding to the previous time n-1 is used as the basis. n-1 Based on the data objects in the cache group for that category, the following formula is used: T n =T n-1 (1−α)+2α / (1+e -α*count ) Calculate the data temperature T corresponding to the current time n for this category of cached group. n , where α represents the preset temperature rise / fall coefficient, e represents the natural constant, and count represents the number of data objects distributed to the cache group of this category.

5. A system for implementing the streaming data processing method according to any one of claims 1 to 4, characterized in that: It includes a data access module, a data aggregation module, a data compression module, a temperature measurement module, a window adjustment module, and a result output module; The data access module is used to execute step A, which enables the data object to be sent to the corresponding category cache group; The temperature measurement module is used to execute step B to obtain the data temperature of each category of cache group at the current time. The data aggregation module is used to perform step C, which aggregates each data object in each category of cache group. The data compression module is used to compress each data object in each category of cache group in step C. The window adjustment module controls the window size based on the data space occupied by each type of cache group and the handling of whether or not each data object in each type of cache group is released in step D. The output module releases each data object in the category cache group that is below the preset lower temperature threshold, according to the steps used to execute step D.

Images