Data stream dynamic fragmentation processing method of real-time calculation scene
By introducing a lease table and lease version number mechanism, the shortcomings of data stream sharding and connection management in real-time computing services are resolved, achieving data processing consistency and system stability, and making it suitable for high-concurrency, high-real-time distributed real-time computing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV OF FINANCE & ECONOMICS
- Filing Date
- 2026-01-19
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies struggle to effectively manage the data stream fragmentation relationship in real-time computing applications with high concurrency, high throughput, and strong consistency requirements, leading to problems such as duplicate data processing, disordered processing order, and partial data loss.
By introducing a lease table and lease version number mechanism, old leases are revoked and new leases with incrementing version numbers are issued. During the transition period, a replication and delivery strategy is adopted to ensure that data is processed in a consistent sharded environment. The lease version stamp, recognition credentials, and sequence number control are used to avoid duplicate processing and loss.
It enhances the ability to continuously process data during sharding switching, ensuring data processing consistency and system stability, and is suitable for high-concurrency, high-real-time distributed computing scenarios.
Smart Images

Figure CN121935290A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for dynamic data stream fragmentation, specifically a method for dynamic data stream fragmentation in real-time computing scenarios. Background Technology
[0002] Currently, although existing technologies have explored aspects such as system architecture, resource scheduling, and sharding strategies for dynamic data stream sharding, it can be seen from the three comparative documents CN104978232A, CN114925073A, and CN110990129A that these solutions still have significant shortcomings and are difficult to fully support complex real-time computing services with high concurrency, high throughput, and strong consistency requirements.
[0003] Firstly, real-time streaming computing scaling solutions, exemplified by CN104978232A, focus on splitting computing tasks and distributing them across more computing units when the load on a computing unit exceeds a threshold, thereby alleviating pressure on a single node and reclaiming resources when the load decreases. These methods primarily address resource elastic scheduling rather than fine-grained management of the data stream's inherent relationships. Their task splitting is based on the task granularity of a distributed message queue, lacking clear mapping and control at the key space or tag set level. Therefore, when the underlying data stream's sharding relationships change, no explicit versioning management mechanism is introduced, nor is corresponding data-level consistency judgment logic provided. This easily leads to problems such as data duplication, disordered processing order, and even partial data loss during task migration or resource expansion—issues unacceptable in businesses with extremely high real-time consistency requirements, such as financial risk control, accounting records, and indicator monitoring.
[0004] Secondly, CN114925073A proposes a distributed database architecture that supports flexible dynamic sharding. Its advantages are mainly reflected in the database storage layer and query execution layer. By dynamically sharding tables and supporting changes in sharding strategies and the addition or removal of replica node groups, it improves the efficiency of accessing large tables. However, this solution is still positioned as a database system. Its dynamic sharding is carried out under transactional consistency and persistence semantics. The adjustment of its sharding metadata largely relies on a centralized management module, and it lacks fine-grained control over the processing path of real-time streaming data, as well as a version stamping mechanism for processing streaming data row by row. Therefore, when the database sharding strategy is dynamically adjusted, it does not provide a clear design for how data is delivered during the transition phase, how to avoid duplicates caused by simultaneous writing to old and new shards, and how to ensure stable output of data according to the processing sequence. Furthermore, it lacks mechanisms for real-time streaming processing characteristics, such as version switching transition states, dual-shard replication delivery, and credential deduplication, making it difficult to directly apply this solution to strictly time-series driven streaming computing engines.
[0005] Secondly, CN110990129A selects suitable storage nodes through an intelligent heuristic algorithm and uses a lease table to control task allocation. This design improves task scheduling efficiency and reduces execution time. However, its leases are mostly used for node-level task scheduling permissions and do not refine the scope of leases to the granularity of key spaces or tag sets and specific data sharding relationships. It also lacks mechanisms such as lease version incrementing, previous lease identification, and revocation credential verification to ensure the correctness of the update chain. Furthermore, it does not attach a lease version stamp or similar information to each piece of data for consistency determination. Therefore, when task reassignment, node online / offline status, or lease switching occurs, risks such as inconsistent task processing semantics and some historical leases still being output may still exist. In addition, all three of the above solutions generally lack a systematic design for handling expired data and sealing / converging mechanisms. Specifically, when an old relationship has not completely exited while a new relationship has taken effect, how to identify old version data, how to prevent old version data from continuing to be output, and under what conditions can data be safely discarded are not systematically addressed. This leads to the system needing to rely on business-side compensation or manual troubleshooting during the version switch window, which significantly increases system complexity and operational risks. Summary of the Invention
[0006] The purpose of this invention is to provide a method for dynamic data stream fragmentation in real-time computing scenarios, thereby addressing some of the drawbacks and shortcomings mentioned in the background art.
[0007] The present invention adopts the following technical solution to solve the above-mentioned technical problems:
[0008] The system maintains a lease table for each shard, which records the key space or tag set and the corresponding lease version number. When data arrives, the routing module delivers the data to the shard corresponding to the key space or tag set based on the currently valid lease version in the lease table.
[0009] When it is necessary to adjust the inheritance relationship of the key space or tag set, the old lease is revoked and a new lease with an incrementing lease version number is generated and published, so that each node updates the inheritance shard relationship of the key space or tag set; during the version switch, the lease corresponding to the key space or tag set to be adjusted is set to a transition state, and the data of the key space or tag set in the transition state is simultaneously copied and delivered to the old shard and the new shard.
[0010] Each piece of data carries a lease version stamp. The processing node compares the lease version stamp with the local lease version: if they are the same, the data is processed and output; if the lease version stamp is less than the local lease version, the data is considered expired and not output; if the lease version stamp is greater than the local lease version, the data is cached and a lease refresh is triggered. The cached data is then processed after the local lease is updated.
[0011] Furthermore, the lease table records the preceding lease identifier and cancellation certificate identifier corresponding to each lease version; when the processing node triggers a lease refresh, the local lease version number is updated only when it is determined that the preceding lease identifier of the new lease is equal to the local current lease identifier, and the cancellation certificate identifier of the new lease is consistent with the cancellation certificate identifier of the cancelled lease.
[0012] Furthermore, during the copy delivery, the routing module carries an acknowledgment credential for the data. The acknowledgment credential consists of a lease version stamp and a monotonically increasing sequence number in the corresponding key space or tag set. The processing node only outputs and records the acknowledgment credential when the lease version stamp is equal to the local version and the acknowledgment credential is appearing for the first time. Data with repeated acknowledgment credentials does not generate a final output.
[0013] Furthermore, when the lease version stamp is less than the local version and is not output as expired data, the processing node queries the routing module to see if the lease version has been sealed; if it has been sealed, the expired data is discarded; if it has not been sealed, the expired data is marked as data to be sealed and a sealing request message is sent to the old fragment. The data to be sealed does not generate the final output.
[0014] Furthermore, the routing module maintains an independent sequence number generator for each key space or tag set and binds the sequence number generator to the lease version stamp; when the lease version stamp is updated, the routing module enables the sequence number generator corresponding to the new lease version stamp for that key space or tag set.
[0015] Furthermore, during the copy delivery, the routing module carries the same acknowledgment credential for multiple copies of data generated with the same acknowledgment credential, and carries a copy identifier in the copy data; after recording the acknowledgment credential, the processing node directly determines the copy data with the same acknowledgment credential but different copy identifiers as duplicate data and does not generate the final output.
[0016] Furthermore, when the processing node receives data where the lease version stamp equals the local version and acknowledges that the sequence number carried in the credential is greater than the maximum consecutive sequence number already recorded locally, the processing node calculates the current maximum consecutive sequence number according to the following mathematical definition:
[0017]
[0018] in: Indicates the maximum consecutively releaseable sequence number; Represents the set of positive integers; This indicates that the processing node has completed the initial occurrence determination and recorded it as the set of accepted data sequence numbers; The upper boundary of the consecutive candidate indices;
[0019] When the sequence number of the received data satisfy When this occurs, it indicates that a gap exists in the sequence number interval [1, s]. The processing node caches this data according to the acknowledgment credential. When the sequence number data corresponding to the gap arrives and the first occurrence determination is completed, the set is... The maximum continuous value in Update: The processing nodes release the corresponding cached data sequentially according to the new continuous boundaries and enter the processing flow. During the release process, the final output is always generated only for the first appearance of the recognition credential.
[0020] Furthermore, the processing node sets a pending-sealing aggregation identifier for unsealed expired data, and sends a sealing request message only once for pending-sealing data with the same pending-sealing aggregation identifier; before receiving the sealing confirmation message returned by the old fragment, it only merges and records subsequent pending-sealing data with the same aggregation identifier without repeatedly sending sealing requests, and none of the pending-sealing data generates a final output.
[0021] Furthermore, the processing node sets a sealing waiting limit for the data to be sealed. If no sealing confirmation is obtained within the sealing waiting limit, the processing node transfers the data to be sealed into the isolation buffer and initiates a sealing status query to the routing module again. When the subsequent query result shows that the data has been sealed, the corresponding data to be sealed in the isolation buffer is discarded, and the data in the isolation buffer never generates a final output.
[0022] Furthermore, the sealing waiting limit is set separately at the granularity of the lease version stamp carried by the data to be sealed; the processing node maintains the corresponding waiting timer for different lease version stamps. When the waiting timer corresponding to a certain lease version stamp reaches the sealing waiting limit and no sealing confirmation is obtained, the data to be sealed corresponding to that lease version stamp is transferred to the isolation buffer and the sealing status query is triggered again.
[0023] The beneficial effects of this invention are as follows: By introducing a lease table and lease version number mechanism, controllable and dynamic adjustment of the key space or tag set inheritance relationship is achieved. During sharding switching, by canceling old leases and issuing new leases with incrementing version numbers, and adopting a replication and delivery strategy in the transition state, data loss, processing gaps, and system interruptions caused by sharding migration are effectively avoided, thus improving the continuous operation capability of the real-time computing system under dynamic load scenarios.
[0024] By carrying lease version stamps for the data, processing nodes can make decisions on processing, caching, or discarding based on version comparisons, ensuring that data is always processed in a sharded environment consistent with its underlying relationships, thus avoiding data misprocessing caused by lease inconsistencies. Simultaneously, dual verification using prior lease identifiers and revocation credential identifiers prevents processing nodes from loading incorrect or invalid leases, enhancing system stability and security.
[0025] By eliminating duplicate processing and output in scenarios involving copy delivery and out-of-order arrival through the credential recognition mechanism, and combining sequence number continuity control and sealing confirmation mechanisms, the system overhead is reduced while ensuring consistency, making it suitable for high-concurrency, high-real-time distributed real-time computing scenarios. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the data processing logic judgment based on the lease version stamp of this invention.
[0027] Figure 2 This is a diagram illustrating the data consistency function of lease inheritance verification and recognition credentials in this invention.
[0028] Figure 3 This is a diagram showing the relationship between the state evolution and cache release of the maximum consecutive sequence number C in this invention.
[0029] Figure 4 This is a schematic diagram illustrating the validity verification of lease refresh in Embodiment 1 of the present invention.
[0030] Figure 5 This is a schematic diagram of deduplication based on acknowledgment credentials in the copy delivery scenario of Embodiment 1 of the present invention.
[0031] Figure 6 This is a schematic diagram of the processing results of expired data under the confirmed sealing state in Embodiment 1 of the present invention.
[0032] Figure 7 This is a schematic diagram of the sequence number generator corresponding to different lease versions and the spatial isolation of the recognition credentials in Embodiment 1 of the present invention.
[0033] Figure 8 This is a schematic diagram of the sequence number continuity determination and cache release process in Embodiment 2 of the present invention.
[0034] Figure 9 This is a schematic diagram of the sealing request suppression mechanism for the expired lease version pending sealing aggregation identifier in Embodiment 2 of the present invention.
[0035] Figure 10 This is a schematic diagram of the independent sealing and isolation process based on lease version granularity in Embodiment 2 of the present invention. Detailed Implementation
[0036] Combined with appendix Figure 1As shown, in one specific embodiment of the present invention, the real-time computing system manages the data stream at a sharding granularity and maintains a lease table for each shard during system initialization or operation. The lease table describes the mapping relationship between key spaces or tag sets in the data stream and the processing shards, and assigns a corresponding lease version number to each key space or tag set to identify the currently valid data connection relationship. The lease version number increments as the shard connection relationship is adjusted, distinguishing the data ownership status at different time stages, thereby providing a basis for subsequent data routing and consistency judgment.
[0037] During data stream processing, when data arrives at the real-time computing system, the routing module first parses the key space information or tag set information carried in the data, and uses this as an index to search for a matching lease record in the lease table. Based on the currently valid lease version number identified in the lease record, the routing module determines the target shard corresponding to the key space or tag set, and delivers the data to the determined shard for processing.
[0038] When the real-time computing system detects that the connection relationship of a certain key space or tag set needs to be adjusted during operation, the system triggers the corresponding lease update process. The adjustment may be triggered by changes in shard load, changes in processing capacity, or changes in system operation strategy. However, regardless of the triggering reason, it is completed through a unified lease revocation and release mechanism to ensure the consistency and controllability of data routing behavior.
[0039] During the adjustment process, the system first revokes the currently effective old leases for the key space or tag set and generates corresponding new lease records. The lease version number of the new lease is incremented from the original to clearly distinguish the time sequence of the old and new lease relationships. After the new lease is generated, it is published to all nodes in the system, enabling each node to gradually update the sharding relationship of the key space or tag set based on the updated lease version number during subsequent data processing.
[0040] During lease version switching, to avoid data loss or inconsistent processing due to changes in shard assignment relationships, the system marks the lease status corresponding to the adjusted key space or tag set as a transitional state. In the transitional state, the routing module simultaneously executes a bidirectional delivery strategy for data belonging to that key space or tag set, copying the data and delivering it to both the old shard corresponding to the original assignment relationship and the new shard indicated by the new lease.
[0041] To ensure the consistency and uniqueness of data processing results during lease switching and dynamic adjustments to sharding relationships, the system attaches a lease version stamp to each piece of data when it enters the processing chain. This lease version stamp identifies the lease version on which the data was generated or routed, enabling processing nodes to clearly determine the version stage of the sharding relationship corresponding to the data.
[0042] When a processing node receives data carrying a lease version stamp, it compares the lease version stamp with the currently active lease version on its local machine. If the lease version stamp is the same as the local lease version, it indicates that the data is consistent with the sharding assumption relationship currently used by the processing node, and the processing node directly performs normal calculations on the data and generates the final output result. If the lease version stamp is less than the local lease version, it means that the lease version corresponding to the data has been updated and replaced, and the data is determined to be expired data. The processing node does not generate a final output for this data to avoid duplicate or erroneous results due to old assumption relationships.
[0043] When the lease version stamp is greater than the local lease version, it indicates that the processing node has not yet completed the update of the corresponding lease version, and the data belongs to a sharding relationship that has not yet taken effect locally. In this case, the processing node temporarily stores the data in the cache area and triggers a lease refresh process to obtain the latest lease information corresponding to the lease version stamp. After the processing node completes the local lease version update, it processes the cached data according to the updated lease sharding relationship, thereby ensuring that all data is processed under a sharding relationship consistent with its lease version, avoiding data loss or incorrect processing order due to lease asynchrony.
[0044] Combined with appendix Figure 2 As shown, to ensure the correctness of the lease update process and prevent processing nodes from loading incorrect or invalid lease versions, the system, in addition to recording the key space or tag set and its corresponding lease version number in the lease table, further records a preceding lease identifier and a revocation credential identifier for each lease version. The preceding lease identifier is used to indicate the previous lease version relationship directly inherited by the current lease version, and the revocation credential identifier is used to represent the valid credential when the corresponding lease is revoked or replaced, thereby forming a verifiable inheritance chain between lease versions.
[0045] When a processing node receives data carrying a higher lease version stamp or triggers a lease refresh operation due to a notification from the routing module during operation, the processing node does not directly update the local lease version number. Instead, it first retrieves the preceding lease identifier and cancellation credential identifier corresponding to the lease to be updated from the lease table. The processing node compares the preceding lease identifier with the identifier of the currently active lease locally, and simultaneously verifies the cancellation credential identifier with the cancellation credential identifier corresponding to the currently active lease locally. Only when it is determined that the preceding lease identifier of the lease to be updated matches the currently active lease identifier, and the cancellation credential identifier of the lease to be updated matches the cancellation credential identifier of the cancelled lease, does the processing node perform the local lease version number update operation.
[0046] To avoid duplicate processing and output when a lease is in transition and data is simultaneously delivered to both the old and new shards, the system generates and carries an acknowledgment credential for each data item during replication and delivery. This acknowledgment credential consists of a lease version stamp and a monotonically increasing sequence number allocated within the corresponding key space or tag set. It is used to uniquely identify the same data or the same processing semantics during the transition period, thus providing reliable deduplication and decision-making criteria for processing nodes.
[0047] When a processing node receives data carrying an acknowledgment credential, it first compares the lease version stamp of the data with the currently active lease version locally. Only if the lease version stamp matches the local lease version will the processing node further check whether the acknowledgment credential has already been recorded locally. If the acknowledgment credential has not yet appeared locally, it indicates that the data is the first arrival data corresponding to the acknowledgment credential. The processing node performs calculations on the data and generates the final output result, while simultaneously recording the acknowledgment credential in the local acknowledgment credential record structure to indicate that the acknowledgment credential has been processed. If the acknowledgment credential carried by the data received by the processing node already exists in the local record, it indicates that the data is duplicate data generated during the replication and delivery process, or a re-arrival of the same data.
[0048] When a processing node receives data carrying a lease version stamp during data processing and determines that the lease version stamp is less than the currently active lease version, the processing node identifies the data as expired data generated based on the old succession relationship. To prevent the accidental loss of historical data that may still affect consistency before the lease switchover has fully converged, the processing node will not immediately discard the expired data, but will first initiate a sealing status confirmation process.
[0049] Specifically, the processing node sends a query request to the routing module to confirm whether the lease version corresponding to the expired data has been sealed. Sealing indicates that the sharding relationship corresponding to the old lease has been officially terminated, and no new final output results are allowed based on that lease version. When the routing module returns a query result indicating that the lease version has been sealed, the processing node confirms that the expired data is no longer meaningful for processing and discards it directly, thereby freeing up processing resources.
[0050] When the query result indicates that the lease version has not yet been sealed, the processing node marks the expired data as data to be sealed and sends a sealing request message to the old shard indicated by the corresponding old lease to prompt the old shard to complete the sealing operation for the lease version. During this process, the data to be sealed is retained but does not participate in the calculation and processing, nor does it generate any final output results.
[0051] In this embodiment, the routing module maintains an independent sequence number generator for each key space or tag set. The sequence number generator is bound to the corresponding lease version stamp and used to assign monotonically increasing sequence numbers to data belonging to the same key space or tag set during the lease version's validity period, thus forming a unique acknowledgment credential together with the lease version stamp.
[0052] When the routing module detects an update to the lease version stamp of a key space or tag set, it no longer uses the sequence number generator corresponding to the original lease version. Instead, it enables a sequence number generator corresponding to the new lease version stamp for that key space or tag set. By switching the sequence number generator during lease version switching, the recognition credentials generated under different lease versions are isolated from each other in the sequence number space, avoiding data confusion across lease versions due to sequence number reuse.
[0053] During the transitional replication and delivery process, the routing module generates multiple copies of data for the same acknowledgment credential, each carrying the same acknowledgment credential and attaching a replication identifier to each copy to distinguish different delivery paths. Upon receiving data carrying the acknowledgment credential, the processing node first checks if it matches the local lease version based on the lease version stamp. If the processing conditions are met, the acknowledgment credential is recorded in the local acknowledgment credential record structure. For subsequently arriving data with the same acknowledgment credential but a different replication identifier, the processing node directly determines it as duplicate data based on the already recorded acknowledgment credential and does not generate a final output for that data.
[0054] Combined with appendix Figure 3As shown, in one specific embodiment of the present invention, after receiving data and completing the lease version consistency determination, the processing node also needs to further ensure the consistency of data processing order in scenarios of replication delivery and out-of-order arrival. Therefore, when the processing node receives data whose lease version stamp is equal to its local lease version, and whose acknowledgment credential carries a sequence number greater than the maximum consecutive sequence number already recorded locally, the processing node first calculates the boundary of the currently releasable consecutive sequence numbers based on the acknowledged data sequence number set. The processing node maintains a sequence number set. The set This is used to record the sequence number of data that has been determined for the first time and confirmed as valid. Based on this, the processing node determines the current maximum consecutive sequence number according to the following mathematical definition. :
[0055]
[0056] in, This indicates that the set starts from index 1. The upper bound of the largest consecutive index in the sequence. Represents the set of positive integers. This represents the set of data sequence numbers that the processing node has currently recognized. These are candidate positive integer values used to determine continuity. Based on the above definition, processing nodes can determine the currently formed continuous sequence interval using set inclusion relationships and extreme value operations.
[0057] The sequence number of the data received by the processing node is denoted as And satisfy When, it indicates that in the sequence number range 1 to There is at least one gap sequence number that has not yet arrived or been acknowledged. At this point, the processing node determines that the data does not yet meet the continuous release condition and stores the data in the cache area according to its acknowledgment credentials, without immediately entering the processing flow. Subsequently, as the sequence number data corresponding to the gap arrives successively and the first occurrence determination is completed, set A is updated, expanding the range of continuously coverable sequence numbers. The processing node then recalculates the new maximum continuous sequence number according to the aforementioned mathematical definition. .
[0058] When the new maximum consecutive sequence number When the sequence number of some or all cached data is increased and covered, the processing node releases the cached data corresponding to the continuous boundary in ascending order of sequence number and enters the processing flow. Throughout the release process, the processing node still follows the first occurrence determination rule of the acknowledgment credential, and only generates the final output for the data corresponding to the acknowledgment credential that appears for the first time and has not yet been recorded in set A.
[0059] In this embodiment, the processing node generates a pending-sealing aggregation identifier for expired data that has not yet been sealed. The pending-sealing aggregation identifier is used to characterize the lease version to which the expired data belongs and the corresponding key space or tag set, so that expired data belonging to the same sealing range can be uniformly merged and managed.
[0060] When a processing node first identifies expired data corresponding to a pending-sealing aggregate identifier, it sends a sealing request message to the old shard, requesting the old shard to complete the sealing operation for the corresponding lease version. After this sealing request message is sent, the processing node no longer sends sealing requests repeatedly for subsequent expired data corresponding to the same pending-sealing aggregate identifier. Instead, it merges and records these subsequent expired data, uniformly marking them as pending-sealing. During the merging process, the processing node can simply record whether there is still pending-sealing data under that aggregate identifier, without triggering the sealing process for each piece of data separately.
[0061] Before the old shard returns a sealing confirmation message, all data marked as pending sealing is not processed and no final output is generated. This avoids duplicate output or inconsistent results based on the old lease before sealing is completed. Once the processing node receives the sealing confirmation message from the old shard, it can perform subsequent discarding or cleanup operations on the expired data under the corresponding pending sealing aggregate identifier based on the sealing confirmation result.
[0062] After the processing node marks expired data as data to be sealed and sends the sealing request, the processing node starts a corresponding waiting timer for the data to be sealed, which limits the maximum allowed waiting time for the sealing confirmation to be returned.
[0063] Within the sealing waiting limit, the processing node continuously listens for sealing confirmation messages from older shards and maintains the pending processing status of the data to be sealed. If no corresponding sealing confirmation message is received within the sealing waiting limit, the processing node determines that there is a risk of delay or anomaly in sealing confirmation. To prevent the data to be sealed from continuing to occupy the normal processing path, the processing node transfers the data to be sealed to an isolation buffer for centralized storage. At the same time, the processing node initiates another sealing status query to the routing module to confirm whether the sealing status of the corresponding lease version has been completed in the system.
[0064] When a processing node learns during a subsequent query that the lease version has been sealed, it immediately discards the data in the isolation buffer corresponding to that lease version that is yet to be sealed, and releases the corresponding storage resources. Throughout this process, the data in the isolation buffer never enters the normal computational processing flow, nor does it generate any final output results.
[0065] In this embodiment, the processing node maintains an independent waiting timer for each lease version with pending data, which is used to record the waiting time after each lease version enters the pending sealing state.
[0066] During processing, when a processing node first marks the data corresponding to a certain lease version stamp as data to be sealed, it initializes the corresponding waiting timer for that lease version stamp and starts timing according to the sealing waiting limit corresponding to that lease version stamp. During operation, the processing node continuously monitors the status of each waiting timer and determines whether the sealing waiting limit for the corresponding lease version stamp has been reached.
[0067] When the waiting timer corresponding to a certain lease version stamp reaches the sealing waiting limit and the processing node has not yet received a sealing confirmation message from the old shard, the processing node only performs subsequent isolation processing on the data to be sealed corresponding to that lease version stamp, without affecting the data to be sealed under other lease version stamps. Specifically, the processing node transfers the data to be sealed corresponding to that lease version stamp to the isolation buffer for storage, and simultaneously triggers another sealing status query to the routing module to confirm the sealing progress of that lease version at the system level.
[0068] Example 1:
[0069] A real-time computing system processes online transaction log data streams. The data is partitioned into key spaces based on user identifiers and distributed to multiple processing nodes for real-time computation using dynamic sharding. Initially, the system maps user identifiers from 1 to 500000 to shard S1, with a corresponding lease version number of V1. The previous lease identifier is empty, and the revocation credential identifier is denoted as R1. At this point, processing node N1 locally loads lease version V1 and continuously processes arriving data based on this lease.
[0070] As the system runs, a continuous increase in processing latency was detected in shard S1. The system determined that the inheritance relationship of key spaces 1 to 500,000 needed adjustment. The control module generated a new lease version V2 to migrate the key space to shard S2. The previous lease identifier recorded in the new lease V2 is V1, and the corresponding revocation credential identifier is recorded as R1, indicating that the new lease is a direct inheritance and replacement of V1. After lease V2 is published to the lease table, the routing module begins to include the lease version stamp V2 in some data.
[0071] During operation, processing node N1 receives data carrying a lease version stamp V2. Since this version stamp is greater than the local current lease version V1, the processing node triggers a lease refresh process. The processing node reads from the lease table that the preceding lease identifier for lease V2 is V1, and the cancellation credential identifier is R1. The processing node compares this preceding lease identifier with the local current lease identifier V1; the result is consistent. Simultaneously, it compares the cancellation credential identifier R1 of the new lease with the cancellation credential identifier R1 corresponding to the local lease V1; the result is consistent. Based on the simultaneous satisfaction of these two verification conditions, the processing node determines that the lease update link is valid, updates the local lease version number from V1 to V2, and continues to process the cached data according to the new lease succession relationship.
[0072] During the aforementioned lease refresh process, the processing node simultaneously verifies the validity of multiple lease versions, with the results as follows: Figure 4 As shown. Figure 4 The document displays the verification results of the preceding lease identifier, the revocation credential identifier, and the final determination of whether the update is allowed for the target lease version V2 and the abnormal lease version V3. Figure 4 As can be seen, when a new lease simultaneously meets the two conditions that the previous lease identifier is equal to the local current lease identifier and the revocation credential identifier is consistent, the processing node allows the lease update to be completed; when either condition is not met, the update is rejected, thereby ensuring the security and consistency of the lease update link.
[0073] During the above process, if any anomalies occur, such as a lease version V3 appearing in the lease table with its preceding lease incorrectly marked as V0, or its revocation credential identifier recorded as R2, the processing node will find that the preceding lease identifier is not equal to the local current lease identifier, or the revocation credential identifier is inconsistent, after comparing the two when triggering a refresh. In this case, the processing node will not update the local lease version number, but will keep the current lease state unchanged, thereby avoiding data being routed to the wrong shard due to loading an incorrect lease.
[0074] In this embodiment, data is divided into key spaces according to user identifiers and is in a copy delivery state during lease switching. In the above embodiment, key spaces 1 to 500000 have been switched from lease version V1 to lease version V2 and are currently in a transitional phase. The routing module needs to deliver data belonging to this key space to both the old fragment S1 and the new fragment S2 simultaneously.
[0075] During the replication and delivery process, the routing module generates an acknowledgment credential for each piece of data. This acknowledgment credential consists of the current lease version stamp and a monotonically increasing sequence number allocated within the corresponding key space. For example, when transaction data with a user identifier of 123456 arrives at the routing module, the lease version stamp corresponding to this data is V2, and the latest sequence number allocated to this key space under lease version V2 is 1008. Therefore, the acknowledgment credential generated by the routing module for this data is V2 plus 1008. Subsequently, the routing module replicates and delivers the data carrying the same acknowledgment credential to both the old shard S1 and the new shard S2 simultaneously.
[0076] Upon receiving the data, both processing nodes S1 and S2 first compare the lease version stamp carried in the data with their respective currently active lease versions locally. In this embodiment, the local lease version of the new shard S2 has been updated to V2, therefore it is determined that the lease version stamp is consistent with the local version, and the process proceeds to the acknowledgment credential determination process; the local lease version of the old shard S1 is still V1, therefore it is determined that the lease version stamp is greater than the local version, the data is cached and a lease refresh is triggered, and it is not included in the final output.
[0077] When the new shard S2 receives data carrying acknowledgment credential V2 plus 1008, the processing node checks its local acknowledgment credential record structure to see if the acknowledgment credential already exists. Since this acknowledgment credential is appearing for the first time, the processing node performs real-time calculations on the data and generates the final output result. At the same time, it records the acknowledgment credential V2 plus 1008 into its local record structure for subsequent deduplication.
[0078] Subsequently, due to the replication and delivery mechanism, data corresponding to the same acknowledgment credential may arrive at the new shard S2 again via other paths, such as due to network retransmission or routing concurrency. At this time, the processing node detects that the acknowledgment credential already exists in its local record structure, directly classifies the data as duplicate data, and no longer generates output.
[0079] The overall processing effect of the above copy delivery and deduplication of recognition credentials is as follows: Figure 5 As shown. Figure 5 Statistics show that during replication and delivery, the new shard S2 generates only one final output for each acknowledgment credential, and subsequent data carrying the same acknowledgment credential are successfully deduplicated; the old shard S1 caches the data arriving during the replication process but does not generate any output, thus avoiding duplicate calculations and duplicate result outputs caused by replication and delivery.
[0080] In this embodiment, after the lease switch, key spaces 1 to 500000 have been switched from lease version V1 to lease version V2. The new shard S2 has completed lease loading and is undertaking the main processing tasks, while the old shard S1 is in the stage of gradually withdrawing from the inheritance relationship. Due to the coexistence of network latency and replication delivery, some data carrying the old lease version stamp may still arrive at the processing node in the system.
[0081] During system operation, a new shard S2 receives transaction data with user identifier 345678. This data carries a lease version stamp of V1, while the currently effective lease version on the new shard S2 is V2. Processing node S2 compares the lease version stamps and determines that V1 is less than V2. Therefore, it identifies this data as expired data generated based on the old succession relationship and does not directly generate the final output for this data. To further confirm whether this expired data may still affect the consistency of lease switching, processing node S2 initiates a sealing status confirmation process.
[0082] Processing node S2 sends a query request to the routing module to check whether lease version V1 has completed the sealing operation. Upon receiving the query request, the routing module checks the current lease table and sealing status records. If the routing module returns a query result indicating that lease version V1 has been sealed (meaning the old shard S1 no longer produces any final output for that lease version), then processing node S2 directly discards the expired data, thereby freeing up computing and storage resources.
[0083] If the routing module returns a query result indicating that lease version V1 has not yet been sealed, it means that the old shard S1 may still contain data that has not been fully processed or fully output. In this case, processing node S2 marks the expired data as data to be sealed and sends a sealing request message to the old shard S1. After the old shard completes processing the in-transit data of lease version V1 and returns sealing confirmation information, processing node S2 can confirm that the expired data is no longer meaningful for processing and ultimately discard the corresponding data to be sealed.
[0084] The processing results of the above expired data under different sealing conditions are as follows: Figure 6 As shown. Figure 6 The data shows that during the pre-sealing phase of lease version V1, processing node S2 marks expired data as pending sealing and retains it. After the routing module confirms the sealing is completed, newly arriving expired data and previously retained pending sealing data are directly discarded, thus ensuring the consistency of system output results during lease switching.
[0085] During system operation, when a new transaction arrives at the routing module, such as a transaction request with user identifier 456789, the routing module first identifies the key space to which the data belongs and calls the sequence number generator bound to the lease version stamp V2 to generate a sequence number. If the current count value of this sequence number generator is set to 2050, then the sequence number assigned to this data is 2051. The routing module combines the lease version stamp V2 and the sequence number 2051 to form an acknowledgment credential, and appends this acknowledgment credential to the data for subsequent copy delivery and deduplication checks.
[0086] During the lease version switching process, if the system detects that the lease version of the key space has been updated from V2 to V3, the routing module will stop using the sequence number generator corresponding to the original lease version V2 and enable a new sequence number generator bound to the lease version stamp V3 for the key space.
[0087] The newly activated sequence number generator assigns sequence numbers from the initial state. Even if the sequence number values assigned under different lease versions overlap, the recognition credentials generated under different lease versions remain completely distinct because the recognition credential is composed of both the lease version stamp and the sequence number, thus achieving natural isolation of the recognition credential space. The isolation effect of the recognition credential generation space under different lease versions is as follows: Figure 7 As shown, even if the serial numbers overlap under different lease versions, the final recognition credentials still remain completely distinct because they contain the lease version stamp, thus avoiding cross-version duplication or confusion.
[0088] In replication and delivery scenarios, the routing module needs to deliver the same data to both the old and new shards simultaneously. In this case, the routing module generates multiple copies of the data for the same acknowledgment credential, carrying an identical acknowledgment credential for each copy, while attaching different replication identifiers to distinguish the delivery path. For example, for data with acknowledgment credential V2 plus 2051, the routing module generates two copies with replication identifiers 1 and 2, delivering them to the old shard S1 and the new shard S2 respectively.
[0089] When new shard S2 receives data carrying acknowledgment credential V2 plus 2051 and a replication identifier of 2, the processing node completes the lease version consistency check and confirms that the acknowledgment credential has not yet been recorded locally. It then performs calculations and generates the final output, while simultaneously recording the acknowledgment credential V2 plus 2051 into its local acknowledgment credential record structure. Subsequently, if new shard S2 receives replicated data carrying the same acknowledgment credential but with a different replication identifier, the processing node queries its local record structure and finds that the acknowledgment credential already exists. Therefore, it directly determines that the data is duplicate data and does not generate a final output for it.
[0090] Example 2:
[0091] A real-time computing system processes online transaction log data streams. Data is partitioned into key spaces based on user identifiers, and during the lease version V2 period, the new shard S2 takes over processing. During replication and delivery, the routing module generates an acknowledgment credential for each data item. This acknowledgment credential consists of a lease version stamp (V2) and a monotonically increasing sequence number within that key space. Processing node S2 maintains a local set of data sequence numbers that have been validated and acknowledged upon first appearance, ensuring consistent output order even when out-of-order arrivals and replication / delivery coexist.
[0092] Let the set of data sequence numbers that processing node S2 has successfully processed and acknowledged at a certain moment be denoted as: ;
[0093] At this point, the processing node calculates the current maximum consecutive releaseable sequence number based on set A. Maximum consecutive sequence number The mathematical definition of is:
[0094]
[0095] in, Represents the set of positive integers. This represents the set of data sequence numbers that the processing node has currently recognized. The upper boundary of the consecutive candidate indices. This indicates that the elements in the set exist consecutively starting from index 1. The largest index value in the set. Substituting into the above definition, we can see that indices 1 to 5 are all included in the set. In the set, the number 6 is not included. Therefore, the calculation result is: ;
[0096] In the above state, processing node S2 receives a new transaction data, which carries a lease version stamp of V2 and a sequence number recorded in its acknowledgment credential. ;
[0097] Since the lease version stamp V2 is equal to the local lease version V2 of the processing node, the processing node enters the sequence number continuity determination process. With the current maximum consecutive sequence number By comparison, we can obtain: ;
[0098] The inequality holds, indicating that there exists at least one unreached or unacknowledged gap number within the interval [1,8]. Combined with the already acknowledged set... It can be seen that the gap numbers are 6 and 7.
[0099] Based on the above determination, processing node S2 does not immediately enter the processing flow for the data with sequence number 8. Instead, it caches the data in the local cache according to its recognition credential V2+8, and does not generate any final output.
[0100] Subsequently, the processing node receives data carrying sequence numbers 6 and 7 in sequence. Assuming that data with sequence number 6 arrives first and completes the initial occurrence determination, the acknowledged set is updated as follows: ;
[0101] Substituting the formula for calculating the maximum consecutive sequence number again, we can obtain the result.
[0102]
[0103] However, since sequence number 7 has not yet been recognized and sequence number 8 does not yet meet the continuous release condition, the processing node continues to maintain the cached state of the data for sequence number 8.
[0104] When the data of sequence number 7 arrives and the first occurrence determination is completed, the set Updated again to: ; Calculate the maximum consecutive sequence number again using the above formula, and we get: ;
[0105] At this point, the processing node checks the data in the cache and finds the sequence number of the cached data. satisfy: This means that sequence number 8 immediately follows the current maximum consecutive sequence number, indicating that there is no gap from sequence number 1 to sequence number 8. Based on this, the processing node releases the data with sequence number 8 from the cache sequentially according to the sequence number continuity principle, enters the processing flow, performs calculations on this data, and generates the final output. Simultaneously, the processing node records sequence number 8 into the set. , make the set Updated to: Throughout the process, the processing node consistently adheres to the rule of recognizing the first occurrence of a credential, only recognizing the first occurrence of a credential that has not yet been recorded in the set. The acknowledgement credentials produce the final output.
[0106] The above sequence number continuity determination process and the maximum consecutive releaseable sequence number For details on how it changes over time, please refer to... Figure 8 As shown, in Figure 8 The table provides the event arrival order and the maximum consecutive releaseable sequence number. The update trajectory and the change curve of the number of cached entries in the cache area can intuitively reflect that when the missing sequence number has not been filled, sequence number 8 will be cached and not released, while sequence number 8 will be released continuously after sequence numbers 6 and 7 arrive in turn.
[0107] In this embodiment, the real-time computing system continues to process the online transaction log data stream. Key spaces 1 to 500000 have completed the switch from lease version V1 to lease version V2. The new shard S2 has taken over the main computing tasks, while the old shard S1 is still in the closing stage of exiting the acceptance relationship. Due to the coexistence of network latency and replication delivery, the processing node S2 continuously receives data carrying the old lease version stamp V1 during operation. This data is all judged as expired data and does not generate final output.
[0108] Within a certain time window, processing node S2 receives three transaction data entries with user identifiers 345678, 345689, and 345701. All three entries carry a lease version stamp V1 and share the same key space. Upon first recognizing that lease version V1 has not yet been sealed, the processing node generates a pending-sealing aggregation identifier, denoted as G1, for the lease version and its corresponding key space. The processing node then sends a sealing request message only once to the old shard S1 for this pending-sealing aggregation identifier G1, requesting the old shard to complete the sealing operation for lease version V1.
[0109] After the sealing request message is sent, processing node S2 will no longer repeatedly send sealing requests for subsequent expired data corresponding to the same aggregation identifier G1. For example, when data with user identifiers 345689 and 345701 arrives, the processing node only associates them with aggregation identifier G1 and merges the records, without sending new sealing requests to the old shard S1. During this stage, all data to be sealed associated with aggregation identifier G1 does not enter the real-time computing processing flow, nor does it generate any final output, thus avoiding a sealing request storm in high-concurrency expired data scenarios.
[0110] The above process for generating aggregation identifiers for expired data and suppressing duplicate sealing requests can be found in [reference needed]. Figure 9 As shown. Figure 9 The document outlines the data relationship and sealing request interaction process between the routing module, processing node S2, and old shard S1. It clearly shows that the same aggregation identifier G1 corresponds to only one sealing request operation, and subsequent expired data is only merged and recorded without triggering the sealing action again.
[0111] Based on this, processing node S2 sets a sealing wait limit for aggregation identifier G1, for example, a wait limit of 30 seconds. After sending a sealing request, the processing node starts the corresponding wait timer and continuously listens for sealing confirmation messages from the old shard S1 during the wait period. If a sealing confirmation message is received from the old shard within the 30-second wait limit, the processing node can confirm that lease version V1 has been sealed and uniformly perform a discard operation on all data to be sealed under aggregation identifier G1, thereby releasing storage resources and completing the convergence of the sealing process.
[0112] If no sealing confirmation message is received within the 30-second waiting period, the processing node determines that there may be a delay or anomaly in the sealing confirmation process. At this point, the processing node transfers all data to be sealed corresponding to aggregation identifier G1 to an isolation buffer for centralized storage and sends another sealing status query request for lease version V1 to the routing module. Data in the isolation buffer never enters the normal processing path and does not generate any final output, thus preventing expired data from occupying the main processing channel for an extended period.
[0113] In subsequent queries, if the routing module returns a result indicating that lease version V1 has completed the sealing operation, processing node S2 can confirm that the old shard S1 no longer generates any output results based on this lease version. The processing node then performs a unified discard operation on the data to be sealed corresponding to the aggregation identifier G1 in the isolation buffer, and cleans up the relevant aggregation records and timing status, completing the final convergence processing of this lease version.
[0114] After completing the switch from lease version V1 to lease version V2, the system triggered further adjustments to a portion of the key space from lease version V2 to lease version V3. As a result, while the new shard S3 undertakes the computation task, the processing node S3 may simultaneously receive expired data carrying lease version stamps V1 and V2.
[0115] During this operational phase, processing node S3 does not adopt a uniform global sealing waiting limit for the data to be sealed. Instead, it sets waiting limits separately for each data type based on the lease version stamp it carries. For example, the system sets a sealing waiting limit of 30 seconds for the data to be sealed corresponding to lease version V1 and 15 seconds for the data to be sealed corresponding to lease version V2. Processing node S3 initializes independent waiting time flags for lease versions V1 and V2 respectively, which are used to record the waiting time after each lease version enters the sealing state.
[0116] Within a certain time window, processing node S3 receives two transaction data entries with user identifiers 401234 and 401245, both carrying a lease version stamp V1. The processing node marks these as pending sealing data and associates them with the waiting timer corresponding to lease version stamp V1. Simultaneously, processing node S3 receives another transaction data entry with user identifier 512345, carrying a lease version stamp V2. This data entry is also marked as pending sealing data, but its waiting timer is maintained independently of the timer corresponding to V1.
[0117] After 15 seconds of system operation, the waiting timer for lease version stamp V2 reaches its set sealing waiting limit, but the processing node still hasn't received a sealing confirmation message from the old shard regarding lease version V2. At this point, the processing node only performs isolation processing on the data to be sealed corresponding to lease version stamp V2, transferring the data with user identifier 512345 to the isolation buffer, and then initiating another sealing status query request for lease version V2 to the routing module. For the data to be sealed corresponding to lease version stamp V1, since its waiting timer has not yet reached the 30-second waiting limit, the processing node does not perform isolation processing and continues to maintain its merged record in the waiting-to-be-sealed state.
[0118] Subsequently, after 30 seconds of system operation, the waiting timer for lease version stamp V1 reached its sealing waiting limit, and the processing node still had not received a sealing confirmation message for lease version V1. At this point, the processing node only transferred the data to be sealed associated with lease version stamp V1, i.e., the data with user identifiers 401234 and 401245, to the isolation buffer and initiated another query request to the routing module regarding the sealing status of lease version V1. Throughout this process, the isolation processing of lease version stamp V2 will not affect the waiting timer and sealing process of lease version stamp V1, and vice versa.
[0119] The above flowchart illustrates the process of independently setting the sealing waiting limit based on the lease version and separately advancing the sealing confirmation and isolation procedures. Please refer to the provided text. Figure 10 As shown. Figure 10 The timeline format is used to show the independent waiting timer and isolation process of lease versions V1 and V2, which intuitively reflects that V2 reaches the waiting limit and enters isolation processing before V1, while the timing and processing of V1 are not affected by V2.
[0120] In a subsequent sealing status query, the routing module returns a result indicating that lease version V2 has completed the sealing operation, while lease version V1 has not. Based on this, the processing node immediately discards the data awaiting sealing in the isolation buffer with lease version stamp V2 and clears the corresponding waiting timer and isolation record. For isolated data with lease version stamp V1, the processing node maintains its isolation status and awaits subsequent sealing confirmation, without generating any final output results throughout the process.
[0121] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A method for dynamic data stream fragmentation in real-time computing scenarios, characterized in that... include: The system maintains a lease table for each shard, which records the key space or tag set and the corresponding lease version number; When data arrives, the routing module delivers the data to the fragment corresponding to the key space or tag set based on the currently valid lease version in the lease table. When it is necessary to adjust the inheritance relationship of the key space or tag set, the old lease is revoked and a new lease with an incrementing lease version number is generated and published, so that each node updates the inheritance shard relationship of the key space or tag set; during the version switch, the lease corresponding to the key space or tag set to be adjusted is set to a transition state, and the data of the key space or tag set in the transition state is simultaneously copied and delivered to the old shard and the new shard. Each piece of data carries a lease version stamp. The processing node compares the lease version stamp with the local lease version: if they are the same, the data is processed and output; if the lease version stamp is less than the local lease version, the data is considered expired and not output; if the lease version stamp is greater than the local lease version, the data is cached and a lease refresh is triggered. The cached data is then processed after the local lease is updated.
2. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 1, characterized in that: The lease table records the preceding lease identifier and cancellation certificate identifier corresponding to each lease version. When the processing node triggers a lease refresh, the local lease version number is updated only when it is determined that the preceding lease identifier of the new lease is equal to the local current lease identifier and the cancellation certificate identifier of the new lease is consistent with the cancellation certificate identifier of the revoked lease.
3. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 1, characterized in that: During the replication and delivery process, the routing module carries an acknowledgment credential for the data. The acknowledgment credential consists of a lease version stamp and a monotonically increasing sequence number in the corresponding key space or tag set. The processing node only outputs and records the acknowledgment credential when the lease version stamp is equal to the local version and the acknowledgment credential is appearing for the first time. Data with repeated acknowledgment credentials does not generate a final output.
4. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 1, characterized in that: When the lease version stamp is less than the local version and is not output as expired data, the processing node queries the routing module to see if the lease version has been blocked; if it has been blocked, the expired data is discarded. If the data is not sealed, the expired data is marked as data to be sealed and a sealing request message is sent to the old fragment. The data to be sealed does not produce the final output.
5. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 3, characterized in that: The routing module maintains an independent sequence number generator for each key space or tag set and binds the sequence number generator to the lease version stamp; when the lease version stamp is updated, the routing module enables the sequence number generator corresponding to the new lease version stamp for that key space or tag set.
6. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 3, characterized in that: During the copy delivery, the routing module carries the same acknowledgment credential for multiple copies of data generated with the same acknowledgment credential, and carries a copy identifier in the copy data; after recording the acknowledgment credential, the processing node directly determines the copy data with the same acknowledgment credential but different copy identifiers as duplicate data and does not generate the final output.
7. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 3, characterized in that: When the processing node receives data whose lease version stamp is equal to the local version and whose sequence number is greater than the maximum consecutive sequence number recorded locally, if there is a gap between the sequence number and the maximum consecutive sequence number, the data is cached according to the acknowledgment credential. When the data corresponding to the gap sequence number arrives and the first occurrence determination is completed, the processing node releases the cached data according to the sequence number continuity for processing, and still only generates the final output for the first occurrence of the acknowledgment credential.
8. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 4, characterized in that: The processing node sets a pending-sealing aggregation identifier for unsealed expired data, and sends a sealing request message only once for pending-sealing data with the same pending-sealing aggregation identifier; before receiving the sealing confirmation message returned by the old fragment, it only merges and records subsequent pending-sealing data with the same aggregation identifier without sending sealing requests repeatedly, and none of the pending-sealing data generates a final output.
9. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 4, characterized in that: The processing node sets a sealing waiting limit for the data to be sealed. If no sealing confirmation is obtained within the sealing waiting limit, the processing node transfers the data to be sealed into the isolation buffer and initiates a sealing status query to the routing module again. When the subsequent query result shows that the data has been sealed, the corresponding data to be sealed in the isolation buffer is discarded, and the data in the isolation buffer never generates a final output.
10. The data stream dynamic fragmentation processing method for real-time computing scenarios according to claim 9, characterized in that: The sealing waiting limit is set separately at the granularity of the lease version stamp carried by the data to be sealed; the processing node maintains the corresponding waiting timer for different lease version stamps. When the waiting timer corresponding to a certain lease version stamp reaches the sealing waiting limit and no sealing confirmation is obtained, the data to be sealed corresponding to that lease version stamp is transferred to the isolation buffer and the sealing status query is triggered again.
Citation Information
Patent Citations
Computation resource capacity expansion method for real-time stream-oriented computation, computation resource release method for real-time stream-oriented computation, computation resource capacity expansion device for real-time stream-oriented computation and computation resource release device for real-time stream-oriented computation
CN104978232A
Distributed storage system scheduling method based on intelligent heuristic algorithm
CN110990129A
Distributed database architecture supporting flexible dynamic fragmentation and implementation method thereof
CN114925073A