Data storage method for artificial intelligence learning mode

Abstract

The invention discloses a data storage method for an artificial intelligence learning mode, and relates to the technical field of computer data storage, and the method comprises the steps: 1, merging a multi-source perception stream into blocks in real time at the edge through monotone serial number writing, so as to provide a replayable time sequence; 2, asynchronous erasure coding is executed on the blocks, Merkel roots are calculated and written into a local cache, and dual guarantee of loss tolerance and integrity is achieved; 3, pushing slices and roots to object storage in sequence according to a network, and calling a time travel interface to solidify an incremental snapshot; 4, the cloud end monitors a snapshot hash event, a serial number chain is written through differential scanning, a gap is reconstructed through slices, an index is refreshed, and continuous consistency is kept; 5, the training process generates a Merkel proof online verification sample, and damaged data are immediately interpolated and repaired and an audit chain is recorded; and step 6, after training is finished, generating a leatherwise list and a frozen root, asynchronously cleaning redundant slices, updating a version table, and finally forming single-fingerprint traceable cost archiving.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data storage, and in particular to a data storage method for an artificial intelligence learning model. Background Art

[0002] With the widespread deployment of edge computing nodes, training data generation has evolved from single-point upload to a parallel influx of sensory streams from multiple sources. Existing systems generally employ a near-real-time archiving model based on device timestamps or file naming conventions: heterogeneous terminals write raw samples to local files, which are then uploaded to central object storage by a batch scheduling task. Offline scripts then perform static erasure coding and hash verification. To avoid frequent small file write amplification, a typical approach is to cache the files at the edge for a certain period of time, aggregate several files into an archive package, and then push the entire file. The cloud executes directory traversal and validation scripts before training, and only after verification is decompression and reading initiated. This type of pipeline can meet basic data integrity requirements in environments with ample bandwidth and stable nodes. However, if a terminal loses connection, a link drops, or a node reboots, the upload order is often disrupted, requiring manual log file comparison to recover missing segments, significantly delaying the training pipeline. What's even more problematic is that, to ensure loss tolerance and recovery, existing systems often configure fixed redundancy ratios for all archive packages and preserve complete parity slices for a long time, resulting in a continuous increase in object storage costs and NVMe write amplification. This is particularly burdensome in deep learning scenarios with continuous iterations and frequent A / B experiments.

[0003] In the distributed artificial intelligence learning model, how to establish nanosecond-level sequential coordinates for multi-source perception streams on the edge side, and achieve continuous consistency and single fingerprint auditability of the entire process from data collection, redundant encoding, snapshot solidification, differential reconstruction to online repair during training and even redundant recovery after training under the conditions of continuous fluctuations in cloud-edge links and random node failures, has become a core problem facing the industry.

[0004] Existing solutions based on batch archiving and offline verification lack chained control of write sequence numbers and real-time Merkle verification. If an edge node reboot or link jitter causes a sequence number gap, the cloud can only locate the missing segment through a full rescan or manual log comparison, with delays ranging from minutes to hours. If the training process is initiated during this time, it will read a broken sample stream, causing gradient explosion or model divergence. Furthermore, a fixed redundancy ratio causes the number of verification slices to grow linearly with the data volume. When training versions are frequently iterated, object storage costs and SSD erase and write cycles increase dramatically, failing to meet both cost-controllable engineering targets and regulatory compliance requirements for post-deletion verification.

[0005] Therefore, there is an urgent need for a unified ontology mechanism that can adapt to bandwidth across the entire edge-cloud link, dynamically adjust redundancy, verify integrity in real time, and automatically freeze and recycle redundant resources after training. Otherwise, it will be difficult to balance timeliness, reliability, economy, and traceability in large-scale deployment. Summary of the Invention

[0006] (1) Technical problems solved In response to the shortcomings of the existing technology, the present invention provides a data storage method for artificial intelligence learning mode, which merges multi-source perception streams into blocks in real time at the edge through monotonic write sequence numbers to provide replayable time series; performs asynchronous erasure coding on the blocks and calculates the Merkle root and writes it to the local cache to achieve dual protection of loss tolerance and integrity; pushes slices and roots to object storage in sequence according to the network and calls the time travel interface to solidify incremental snapshots; monitors snapshot hash events in the cloud, differentially scans the write sequence chain, uses slices to rebuild gaps and refreshes the index to maintain continuous consistency, and finally forms a cost archive with traceability of a single fingerprint; thereby solving the technical problems recorded in the background technology.

[0007] (2) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: Data storage methods for AI learning models include assigning monotonic write sequence numbers to multi-source data streams at the edge high-speed absorption layer and merging them into sample blocks in real time, providing reliable timing for subsequent erasure coding and reconciliation, and ensuring fault replayability. Perform asynchronous erasure coding on the sample block based on the write sequence number to generate cross-node redundant slices, calculate the Merkle root and write it into the local cache for integrity verification and differential scanning; The edge side pushes the slices and Merkle roots to the object storage in the order of the write sequence through the bandwidth-aware replication pipeline, and calls the time travel interface to solidify the incremental snapshot and record the backtracking mark; The cloud training scheduler monitors snapshot events to trigger differential scanning, compares the write sequence number chain, and if a gap is detected, reconstructs the missing segment using the redundant slices and refreshes the time travel index to maintain consistency; When replaying samples during the training process, a Merkle proof is generated for each sample to verify integrity. If the verification fails, the damaged sample is marked and repaired using prediction consistency interpolation, and the repair event is recorded at the same time. After training is completed, the snapshot is frozen in the write silent window to generate a history record and lock the frozen root. Then, redundant slices are asynchronously deleted to release cache space, and the object storage version table is revised to generate the final audit root.

[0008] Furthermore, the write sequence number chain is generated by a two-stage write sequence number function determined jointly by the logic clock frequency and the carry window size. The write sequence number maintains global monotony increase in a multi-threaded environment and avoids wraparound by window folding before the count overflows. It is bound to the block merge cache to guide the windowed splicing of multi-source perception streams. The block merge cache has a limited capacity and is cyclically dequeued in the order of the write sequence numbers.

[0009] Furthermore, after the sample block is generated, an asynchronous erasure coding queuer is called, the queuer calculates the priority according to the arrival interval of the merged block and the historical missing block rate, writes the priority into a preemptive priority queue, and inputs the block with the highest priority into the erasure encoder; Timestamp indexes and Merkle root summaries are appended at the slice layer and written to the local cache in a key-value manner to ensure atomic consistency of blocks and metadata. The key-value pairs are stored in a reserved transaction area of ​​a high-speed solid-state cache.

[0010] Furthermore, asynchronous erasure coding uses a Vandermonde matrix with an adjustable redundancy ratio, dynamically adjusts the number of data slices and the number of check slices, and uses a bandwidth-aware slice distribution algorithm to distribute the generated slices to multiple nodes with exponential probability; The coding queuer uses arrival interval, missing block rate and link margin as parameters for real-time sorting to ensure that data slices are written to high-bandwidth nodes first and check slices are elastically distributed with the load. The multiple nodes include edge nodes and central nodes.

[0011] Furthermore, after erasure coding is completed, the block-level root is calculated through the hierarchical Merkle tree and a sub-root hierarchical index is generated. The write sequence number and the timestamp index are concatenated as the key value to write the slice and the Merkle root atom into the local solid-state cache; The key value is a unique serial number timestamp string for subsequent differential scanning and lake warehouse time travel positioning. The solid-state cache uses non-volatile storage and supports direct key addressing, and provides a block root index for subsequent differential scanning.

[0012] Furthermore, the replication pipeline adjusts the sending rate in real time according to link utilization, buffer depth and available bandwidth through an adaptive rate control law, and the pipeline confirmation window limits the confirmation order based on the maximum number of in-flight chips; Any out-of-order confirmation will only slide as a whole after the smallest unconfirmed sequence number is acknowledged, ensuring that the push order is strictly consistent with the write sequence number and preventing reverse overwriting.

[0013] Furthermore, when calling the Hucang time travel interface to solidify the incremental snapshot, the number of snapshot segments is determined by the square root balance formula based on the write sequence span and version span; A traceable marker containing the block root, version boundary and cyclic redundancy check code is inserted at the end of each segment and a segment-level hash is generated so that differential scanning can directly compare the marker to determine the integrity of the segment without the need for directory traversal. The segment-level hash result is written to the incremental snapshot metadata area.

[0014] Furthermore, the differential scanner first calculates the segment-level hash value for the traceable marker and the end-of-segment Merkle root. If the hash does not match, it then builds a binary index structure consisting of a Bloom filter and a jump table with eight sequence numbers to locate the write sequence number gap within two memory accesses; The reconstruction priority queue is calculated based on the gap length and the time aging coefficient for the reconstruction engine to call. The gap priority queue is updated in real time in the memory.

[0015] Furthermore, fast reconstruction uses the slice location vector to pull the minimum set of data slices that meet the recovery conditions at one time, schedules the Tensor Core to perform inverse Vandermonde matrix operations to restore the missing blocks, and uses the incremental grafting algorithm to recalculate only the affected path nodes of the global Merkle tree; At the same time, appending new block key values ​​to the time travel index keeps queries consistent with the snapshot, which involves the row consistency of the time travel index.

[0016] Furthermore, the online verification is performed by the central processing unit (CPU) after the leaf hash is calculated by the tensor core, and the proof path is managed with a fixed length of one kilobyte in the ring buffer. If the verification fails, a damage mark containing the write sequence number, timestamp and first-layer mismatch hash is generated and written into the queue to be repaired. The main training thread ignores damaged samples in batches through masks to maintain the tensor shape, and the damage mark is stored in the shared memory segment of the lock-free ring buffer.

[0017] Furthermore, the interpolation repair algorithm selects the nearest neighbor samples in the same batch and combines them with the model sliding mean to generate a replacement tensor, and monitors the gradient perturbation rate in real time; if it exceeds the preset stability threshold, it automatically increases the mean weight and re-interpolates, and writes the repair entry containing the write sequence number, replacement tensor summary and gradient perturbation into the repair log and updates the repair log root, which is synchronized to the upper node of the global Merkle tree.

[0018] Furthermore, a freeze vector is formed by monitoring the three indicators of active write latency, buffer residual depth and refresh completion rate. When the silent threshold and buffer threshold are met and the refresh completion rate is 100%, a history list sorted by write sequence is generated to calculate the history issuance root, and the snapshot root, repair log root and history issuance root are salted and hashed to generate a freeze root and broadcast to lock the write permission. The history list is securely written to the immutable object storage bucket.

[0019] Furthermore, after the frozen root is generated, data slices of at least the minimum number of data slices are retained according to the retention probability function, and exponential decay deletion is performed on the parity slices. During the deletion process, the erase rate is controlled by write amplification negative feedback scheduling to extend the life of the solid-state medium. After completion, the object storage version table is revised to generate a cleanup root, which is hashed together with the frozen root to obtain the audit final root, and the audit final root is written into the metadata index of the frozen segment.

[0020] (3) Beneficial effects The present invention provides a data storage method for artificial intelligence learning mode, which has the following beneficial effects: Based on the write sequence number chain and block merge cache, nanosecond-level unambiguous timing coordinates are established on the edge side, so that multi-source perception streams can be accurately replayed in the event of any node failure, link jitter or device restart. The sorting cost is moved from the cloud to the collection end, directly promoting the subsequent asynchronous erasure coding into a purely parallel matrix operation path, significantly shortening the write path and reducing the double overhead of sorting and verification.

[0021] Asynchronous erasure coding atomically binds the adjustable Vandermonde matrix, bandwidth-aware slice distribution, and layered Merkle tree. Redundant slices and integrity fingerprints are coupled when each block is generated. The redundancy ratio is adjusted in real time with the link margin without pausing writing. Layered hashing allows local rewrites to only recalculate two layers of nodes, thus maintaining zero additional latency for integrity verification under high concurrent write loads, achieving simultaneous loss tolerance and real-time verification.

[0022] The bandwidth-aware replication pipeline dynamically adjusts the sending rate based on link utilization and buffer depth, and the pipeline confirmation window ensures that slices and Merkle roots are entered into the table strictly in the order of write numbers; the lake warehouse side solidifies incremental snapshots through snapshot segmentation functions and traceable tags, and segment-level hashing enables differential scans to determine integrity without traversing the entire table. The cloud edge has symmetrical views within the sub-second time window, and upload-snapshot-verification enters a self-driven closed loop, completely eliminating the bottleneck of manual reconciliation.

[0023] The cloud-based differential scanner uses segment hashing for coarse screening and Bloom table checking for fine-grained checking to precisely locate write sequence number gaps during constant-level memory access. The fast reconstruction engine uses the slice positioning vector to align the minimum data slice set at once, employs the Tensor Core inverse matrix core to reconstruct missing blocks in milliseconds, and uses a grafting algorithm to refresh only the affected paths in the Merkle tree. The entire process from gap discovery to closure is less than the single-batch forward time, achieving continuous and consistent dynamic self-healing.

[0024] During the training phase, online Merkle proof uses the idle time slots of the Tensor Core to calculate leaf hashes, circular path buffering eliminates DMA fragmentation, and the main thread seamlessly completes sample-level integrity verification. Damaged samples are immediately pushed to the repair queue and replacement tensors are generated using neighbor prediction consistency interpolation. Gradient stabilization gating ensures that the perturbation does not exceed the threshold and writes the repair entry to the repair log chain, achieving both training throughput and accuracy.

[0025] After training, the freeze scheduler generates a history list in the write silent window and combines the snapshot root, repair root and history root into a frozen root through a sealing function. Then the redundant recovery strategy only retains the minimum recoverable slice and deletes the check slice according to the exponential decay probability. Write amplification feedback smoothes the erase rate to extend the life of the solid-state media. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure is a flow chart of the data storage method for artificial intelligence learning mode of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0028] See also Figure 1 The present invention provides a data storage method for artificial intelligence learning mode, comprising: In the AI ​​training scenario driven by multi-source perception streams, the edge high-speed absorption layer Three challenges must be addressed: First, the original samples sent by heterogeneous terminal devices are asynchronously distributed in time. Without a unified order, subsequent erasure coding cannot guarantee the correspondence between block-level data; second, edge nodes often face link jitter and sudden failures. Without a robust sequential replay mechanism, any single point failure will be amplified into a cascading replay error, directly destroying the temporal consistency of the training data. Secondly, cloud-edge collaboration requires continuous output of reliable sample blocks with timestamps in a high-concurrency environment. If the sequence control granularity is not fine enough, it will make it difficult to reconcile the lake warehouse snapshots on demand.

[0029] Step 1: On the edge, a monotonically increasing write sequence number is assigned to heterogeneous real-time streams. These are then merged and concatenated into fixed-granularity sample blocks in a high-speed ring cache. Window start and end pointers are also generated. This provides a nanosecond-level, unambiguous timing coordinate reference for subsequent erasure coding, differential reconciliation, and fault replay, ensuring rigorous traceability of the link. Step 101: Use the write sequence chain Multi-source perception flow The arrival events are monotonically numbered and cached in blocks Aggregate consecutive sequence number samples in real time to establish a block-level time series coordinate system that can be replayed, corrected, and reconciled; Due to the number of edge terminals Exponential growth, the mixing of raw inbound sequences can easily lead to subsequent erasure matrix To this end, the arrival jitter is smoothed through the three-step linkage of sequence number allocation, block-level alignment, and persistent mapping, ensuring that the sample blocks at any time have a monotonically traceable timing benchmark; thus, ensuring that the erasure coding only assumes the function of loss tolerance in the column dimension and no longer corrects the order; and then, when a node failure occurs, the replay pointer The missing section can be directly located by the sequence number gap, and the fault impact domain can be constrained to the minimum block window. Inside.

[0030] High-speed absorption layer at the edge Design logic clock and carry window size The two-segment write sequence number generation function:

[0031] Among them: logical clock : Counting frequency, value range , used to map physical time into discrete ticks; carry window size : The window length is aligned with the block size, which restricts the sequence number to monotonically increase within a single window and prevents 64-bit overflow; Write sequence chain : In any A monotonically increasing identifier of the moment, ensuring that rollback does not occur due to concurrent writes by multiple threads.

[0032] Through the dual constraints of logical clock and window, the sequence number generation only presents an increasing feature externally, while still having fine-grained time resolution internally, which can be directly used for subsequent Merkel summary positioning.

[0033] Designing a Circular Buffer , and define the merge operator:

[0034] in: : Splicing function to ensure that the sequence metadata is bound to the original payload atom; , by writing the serial number Fixed-granularity data unit formed by splicing perceptual fragments; , with the fragment Synchronously generated global monotonically increasing sequence number; , No. A raw perception data frame, which can come from radar, camera or inertial navigation, and has a variable byte length; Block-level XOR merging , which means pressing Dequeue in ascending order; Perform byte-level bitwise XOR on the splicing results one by one, and output a synthetic byte stream of the same length as the single splicing result Block Merge Cache :capacity A sample block is responsible for temporarily storing the consecutive sequence number samples of the same window; Merge operator :For the current sample block The result of the merge operation is a fixed-length byte sequence, which is used as the input for subsequent leaf hashes. Merge limit : The number of fragments in the current block determines the final value of the summation index; By windowing and merging, the erasure matrix can be significantly reduced The number of rows can be increased to improve the efficiency of parallel SIMD coding and provide a fixed-granularity verification unit for the comparison engine.

[0035] To support fault domain Internal precise backtracking, introducing replay pointer:

[0036] in, For the The globally monotonically increasing sequence number of the sample fragments assigned in step 1; A set of confirmed persistent sequence numbers, a set of successful ACK sequence numbers returned by the lake warehouse reconciliation, which grows dynamically; a replay pointer : Points to the first missing sequence number; missing judgment domain : The serial numbers of all unsuccessful reconciliations.

[0037] The pointer locks the gap with O(1) search complexity, and combined with the merge cache, the corresponding sample block can be quickly replayed to achieve millisecond-level data self-healing. By introducing the closed-loop control of the write sequence chain, block merge cache and replay pointer in the edge high-speed absorption layer, a linearly consistent time sequence order without global lock is achieved in the multi-source heterogeneous perception flow scenario, eliminating the block-level misalignment caused by terminal arrival jitter, so that the subsequent erasure matrix only needs to focus on loss tolerance rather than order, significantly reducing the complexity of redundant coding; at the same time, the monotonic sequence makes any node failure appear only as a sequence number gap, which is replayed by the pointer. Capture quickly to converge the recovery range to the minimum block window.

[0038] Step 102: Index with timestamp block As a link, the blocks are merged and cached The blocks in the pipeline are mapped to the asynchronous erasure coding pipeline, ensuring that the block content and timestamp-sequence number metadata are inseparable, and achieving the natural order consistency of erasure coding and integrity verification.

[0039] When a single large piece When entering the erasure encoder, if there is no timestamp index, it will be difficult for the reconciliation engine to quickly locate the corresponding snapshot row in the lake warehouse time travel interface. Therefore, this step inserts a timestamp-sequence number double index into the three-stage flow of block-level erasure coding-Merkel root proof-temporary cache disk. , ; This eliminates the reallocation overhead between redundant slices and local blocks across nodes; thereby maintaining highly sequentially consistent push behavior even when bandwidth is limited; and allowing the cloud-based training scheduler to scan the differences directly by index chain; ultimately achieving seamless spatiotemporal alignment of the edge view and the main lake view.

[0040] In order to perform erasure without blocking the absorption layer, a priority queue based The queuer, whose scheduling priority function :

[0041] Among them: Merge priority : represents a sample block The priority score in the asynchronous erasure coding queue, the larger the value, the higher the priority. Inter-arrival interval : The time difference between two adjacent merged blocks arriving at the queuer; block loss rate : The missing block rate of the source node sample in the historical window, range ; Weight coefficient , :satisfy , dynamic adaptive, initial , ; This function takes into account both real-time performance (arrival interval) and reliability (block missing rate), so that the queuer prioritizes protecting volatile blocks when the link is limited, and the weighted logic avoids coding starvation caused by a single indicator deviation; for each slice after erasure coding, Constructing hash leaves , and then compute the Merkle root bottom-up:

[0042] Among them: root calculation operator , refers to the recursive process of pairing the input hash lists together according to the Merkle tree rule and hashing them layer by layer until a single root value is obtained; is the total number of slices; slice : erasure coding output fragment; timestamp index; Write sequence number chain; hash function : BLAKE3, output 256 bits; Merkle root : Full-film integrity fingerprint, used for subsequent playback verification.

[0043] Write sequence number chain Indexed with timestamp Solidify the hash leaf to ensure that any replay or splicing operation must match the sequence number and timestamp dual index, eliminating the reordering attack surface under hash collision.

[0044] Will Write to solid state cache in the form of key-value pairs , the keys are:

[0045] Of which: Solid-state cache : NVMe array, key : Base64-encoded serial number-timestamp string to ensure uniqueness; value : Contains the block payload and the Merkle root.

[0046] Key uniqueness enables differential scans to locate blocks with only a hash index. Combined with the time travel interface, the differences between edge snapshots and cloud snapshots can be quickly compared.

[0047] The global monotonic write sequence number chain constructed in step 101 -Block Merge Cache -Replay pointer The ternary closed loop, this solution has laid a traceable sequence coordinate for the multi-source perception flow; and step 102 further indexes the timestamp With Merkel Root Deep binding to erasure slices achieves atomic coupling of data integrity and sequential consistency. The two steps are logically sequential: the continuous blocks output by the former and its serial number chain, which directly serves as the input for the latter to generate redundant slices and Merkle roots; the latter in turn uses the Merkle root to reversely verify the continuity of the serial number chain, forming a closed-loop correction.

[0048] Atomically binding the timestamp index and the Merkle root during erasure coding makes integrity verification and sequential consistency naturally homologous, fundamentally avoiding the classic paradox of recovering blocks but failing to align with snapshots. The asynchronous erasure coding queuer driven by a priority queue dynamically balances real-time performance and reliability through priority, automatically protecting volatile blocks in bandwidth-constrained situations. This strategy breaks free from the rigid constraints of traditional static redundancy ratios. Furthermore, the local solid-state cache uses a unique mapping based on sequence number and timestamp keys, eliminating the need for complex directory traversals for differential scans to complete lake warehouse reconciliation, providing a lightweight, traceable snapshot path for ultra-large-scale edge nodes.

[0049] In step 1, write the sequence number chain , block merge cache Indexed with timestamp For each sample block It lays the foundation for unambiguous and traceable sequential coordinates; however, only when these blocks are further converted into cross-node redundant slices and accompanied by tamper-proof integrity fingerprints, can the entire cloud-edge training pipeline truly resist multiple failures such as node loss, disk damage, and link jitter.

[0050] Step 2 is carried out in this context: on the one hand, it is necessary to write the sample block without blocking the absorption layer. Perform asynchronous erasure coding to generate a mixed set of data slices and check slices , so that when a node fails, it can still be used by any The fragment quickly recovers the original block; on the other hand, it needs to synchronously calculate the Merkle root , and key the slice-root pair Write solid-state cache , providing a single source for subsequent differential scanning and auditing. This process not only fully utilizes the parallel encoding capabilities of edge nodes, but also ensures the atomic persistence of metadata and payloads, avoiding the chain coupling delays caused by the traditional encoding-then-verification approach.

[0051] Step 2: While maintaining zero blocking of the write thread, priority queue scheduling is used to perform adjustable redundancy erasure coding on the ordered blocks and simultaneously calculate the Merkle root. The slices and roots are atomically written to the cache as key-value pairs, taking into account both node loss tolerance and block-level integrity verification. At the same time, cross-node redundancy, hash index, and timestamp dual-index mapping are prepared for subsequent bandwidth-aware replication and differential scanning. Step 201: For each sequential sample block , build an asynchronous erasure coding pipeline to output a set of redundant slices across nodes while maintaining zero blocking in the absorption layer , and dynamically adjust the number of data slices according to the network situation and the number of check pieces ; The output rate of the absorption layer is much higher than the single-node storage bandwidth. If the erasure coding is synchronized directly, the write path will be lengthened and the write sequence chain will be destroyed. Therefore, the encoding needs to be placed in an asynchronous thread pool and used as a priority queue. Perform rate shaping to ensure that the writing thread is only responsible for pushing sample blocks To the encoding buffer , while the actual matrix operations are completed in the background; and then with the help of adjustable redundancy When bandwidth is tight, the proportion of parity slices is reduced; then the slices are evenly distributed to different nodes using the Rabin-Karp rolling hash algorithm to avoid single-machine hot spots; finally, even if multiple nodes fail, as long as any The original block can be restored by simply retrieving the original block, thus achieving controllable damage tolerance.

[0052] In order to quickly switch the redundancy level under different loads and failure probabilities, a parameterized Vandermonde matrix is ​​used. , where the redundancy ratio Instantly Adjustable:

[0053] Including: number of data pieces : Value range , the minimum number of recoverable pieces required for encoding; the number of check pieces : Value range , used to resist node failure; redundancy ratio : Value range , indicating the level of damage tolerance redundancy; : Galois field multiplication primitive, the value is primitive elements in; erasure matrix : Determines the generator matrix of the linear combination coefficients of the slices.

[0054] By adjusting the redundancy ratio in real time , the same encoding core can be used to switch redundancy in milliseconds, avoiding the traditional solution of pausing writing due to matrix reconstruction. After the slice is generated, it must be distributed to multiple edge nodes. If it is only written in a polling manner, it is easy to cause the bandwidth bottleneck node to become a single point of recovery. The Rabin-Karp rolling hash algorithm is based on the real-time link margin. Make an exponential decay selection:

[0055] in: Indicates that the source node To the target node Slice transmission probability; slice :Indicates the source nodes, from the sample block Any fragment of; candidate node :Indicates the target nodes, storage nodes in the edge cluster; link margin : The available bandwidth from point to point with the encoding node; Represents the target node The bandwidth of freedom, is the loop index; the temperature coefficient : , adjust the distribution slope, the default is 0.3; This soft distribution strategy tilts writing more slices on high-bandwidth nodes while maintaining probabilistic balance to avoid forming fixed hot spots. The generation can be delayed by the background refresh thread, as long as it is written before the lake warehouse time travel reconciliation is triggered:

[0056] Where: Snapshot deadline : The latest time point at which the current incremental snapshot needs to be completed; encoding completion time : The time when data slice generation is completed; the maximum delay threshold : Used to limit background delay; delay window : The maximum tolerable delay for the background thread to generate the parity slice.

[0057] Among them, delayed generation of check slices can release coding computing power during peak hours and smooth the redundant overhead to the link idle period.

[0058] In asynchronous pipeline mode, the triple mechanisms of adaptive erasure matrix, bandwidth-aware distribution, and background verification slice refresh are introduced to ensure that the write thread of the edge node always remains in a non-blocking state. Even if link jitter or node loss occurs, the dynamic ratio of the number of data slices + the number of verification slices and the probabilistic slice drop strategy can be used to stably limit the minimum set of slices required for recovery to within the preset threshold, demonstrating the innovative ability to maintain linear scaling in an environment with fluctuating resources. At the same time, since the background thread can delay the generation of verification slices, business timeliness can be prioritized during peak write periods, and redundancy can be supplemented during idle periods, thereby balancing energy consumption and hardware wear throughout the entire life cycle.

[0059] Step 202: For each slice set Generate Merkle roots on the fly , and press the key Atomically write slice-root pairs to the solid-state cache , thereby integrating integrity checking with snapshot indexing.

[0060] Redundant slices alone are not enough to guarantee end-to-end trust during the training phase. If any slice is tampered with or misordered in the link, recovery may be successful but the semantics have changed. Therefore, it is necessary to hash each slice together with the order metadata of the block to which it belongs; thus, any single-byte perturbation is mapped to the Merkle root. Change; then the root and key They are all persisted together to ensure that the root can be found by looking up the key; then, on the lake warehouse side, only the root needs to be compared to determine block-level consistency; finally, large-scale differential scanning is completed with minimal bandwidth.

[0061] To ensure that the sample time series attributes are embedded in the integrity fingerprint, the serial number chain is written Indexed with timestamp Spliced ​​into each slice hash preimage, together to ensure that any rearrangement will lead to root changes, eliminating out-of-band replacement attacks:

[0062] Among them: leaf node :slice Hash value of ; hash function :BLAKE3, output 256bi; concatenation operator : Byte order splicing operation.

[0063] In the number of slices When it is large, a single-layer Merkle tree will cause the root update cost to increase Growth; the solution uses a two-level grouping tree to reduce recalculation overhead:

[0064] Including: number of groups : , divide the slice into subgroups; within-group size : The number of slices contained in each group; group root : No. Subroot of a group; global root :piece The final Merkle root is used for cloud-edge consistency verification.

[0065] Among them, the hierarchical tree allows only the root of the group to be recalculated when a local slice is rewritten, and then two layers are updated from top to bottom, which significantly reduces the computing power consumption of low-frequency recalculation. With slice indexing Write to solid-state cache together When , a symmetric two-phase commit is used to prevent partial failures:

[0066] Among them: pre-written : First write to the temporary page and mark it as unsealed; seal : Refresh the index after NVMeFTL is marked as persistent; rollback : Delete unfinished entries; key : Base64 string, unique identification block ; The two-phase commit ensures that even if a power outage occurs in the intermediate state, there will be no inconsistency such as losing the root but retaining the slice or the stub missing the slice, anchoring the minimum transaction unit that can be replayed by the Hucang time travel interface.

[0067] By directly embedding the write sequence number chain and timestamp index into each hash original image, step 202 completes the integrity binding at the moment of slice generation, making the sequence, i.e., the fingerprint, a natural attribute; and then using the layered Merkle tree to reduce the recalculation scope of local rewrites, and using the key A two-phase commit is completed for unique anchoring, ultimately achieving a minimal transactional snapshot of the key, root, and shard trinity. This not only allows differential scans to assert block-level consistency simply by comparing the root, but also flattens the traditional separate, serial process of first performing data and then verifying data into a single, simultaneous process, significantly shortening the reconciliation window.

[0068] Step 2: Through step 201, the edge node can write the sequence number chain without blocking Under the premise of , a set of redundant slices that change in coordination with the network status is output in real time; then step 202 further submits a ternary closed loop with sequence number-timestamp mixed hash-hierarchical Merkle tree-key-value atomic submission to bind the integrity fingerprint and snapshot index into a single transaction.

[0069] The slices produced by the former Directly enter the hash leaf generation of the latter, and the root calculated by the latter This in turn provides a recalculated verification path for the asynchronous verification slice refresh. Compared to the traditional full block redundancy followed by offline verification, the verifiable fingerprint is included during slice generation, creating a synergistic effect of redundancy and verification.

[0070] Following step 2, the slice-root pair is cached in the local solid state. After the atoms in the data are persisted, these verified and bound data units must be safely and efficiently migrated to the cloud-side object storage to generate incremental snapshots that can be referenced by the lake warehouse time travel interface and provide a reconciliation benchmark for subsequent differential scans. Since the access network where the edge node is located is in a mixed state of bandwidth mutation and latency jitter all year round, if it is still pushed in the traditional static batch upload mode, it will not only easily cause write amplification, but also destroy the write sequence number chain due to sequence drift. Monotone traceability property.

[0071] Step 3: Push the slices and Merkle roots to the object storage in the order of the write sequence through the adaptive rate replication pipeline, and use the pipeline confirmation window and global version counter to ensure multi-threaded order preservation; then call the lake warehouse time travel interface to solidify the incremental snapshots in diagonally balanced segments and write traceable marks, so that the edge view and the main lake view are consistent within sub-seconds and provide segment-level fingerprints.

[0072] Step 301: In the case of dynamic bandwidth fluctuation, build an adaptive replication pipeline to make the slice With Merkel Root Able to write serial number The order is pushed to the object storage continuously, and the RTT constraint on throughput is reduced through the pipeline confirmation window.

[0073] Instantaneous available bandwidth of edge-cloud links and queuing delay There is a strong correlation. When the replication rate is too fast, queue congestion will cause long tail delays and out-of-order retransmissions; conversely, peak bandwidth is wasted. Therefore, it is necessary to calculate the utilization of the input link in real time. Local copy buffer depth To adjust the replication rate ; thus automatically entering the conservative window when the load is high and the detection window when the load is low; and then coordinating with the sequential confirmation window Ensure that the order of arrival of replication ACK does not destroy the write sequence number chain; then use monotonic version index on the object storage side The serial number sequence is followed to avoid reverse overwriting of the previous write and the next version number. Even if the link drops sharply from 5Gbps to 200Mbps, the replication rate and sequence control can be stabilized in coordination without manual intervention.

[0074] Replication rate Link utilization and buffer depth For input, high throughput and low disorder are maintained through a dual-ring adaptive function:

[0075] Available bandwidth : The link idle bandwidth measured immediately, range , used as the upper bound of the rate; Link utilization : Real-time ratio, dimensionless, 1 indicates congestion; adaptive coefficient : , adjust the Sigmoid convergence slope; buffer depth : The number of bytes currently waiting to be sent in the copy buffer, range ; Maximum buffer : Fixed upper limit, typically 512MB; suppression coefficient : , determines the reduction of the buffer level on the rate; the replication rate : Current pipeline data transmission rate, in Gbps.

[0076] The dual-factor coupled suppression makes the rate drop rapidly and smoothly once it approaches saturation, and quickly releases bandwidth when the buffer is low and the idle time is high, maintaining the queue isochronous stability; in order to reduce RTT jitter, the window size is used The window slides by the smallest unconfirmed sequence number Advance.

[0077]

[0078] Among them: Maximum window : Pieces, limited to the number of flying pieces; the maximum number of sent sequence numbers : The maximum write sequence number that has been pushed but not necessarily confirmed; the minimum unconfirmed sequence number : Left edge of the window, moves right after ACK arrives; window size : The number of unconfirmed fragments allowed.

[0079] The window ensures that the write sequence chain will not be disrupted even if ACKs arrive out of order. The entire chain will slide only when the leftmost gap is confirmed, reflecting the principle of sequence priority. The object storage version table uses an ascending index. Mapping to keys , the index generation function relies on the global atomic counter :

[0080] Among them: atomic counter : The version number of the last successful write to the object storage; the current version index : The incrementing number generated by this write, 64 bits, guaranteed monotonic; key : , uniquely identifies a slice-root pair.

[0081] The write sequence number is inherited by the global monotonic version number, so that the object storage is naturally order-preserving; even if multiple threads put concurrently, there will be no overwriting or reverse order visibility.

[0082] The adaptive replication pipeline integrates transient control of replication rate, order protection of pipeline confirmation window and persistent mapping of monotonic version index, breaking through the inherent limitation of traditional constant upload speed + batch ACK that is prone to retransmission storm in narrow bandwidth sudden drop scenario. and buffer depth The dual feedback of the system can form a light detection and fast convergence link self-healing rhythm independently without relying on external congestion protocols; and the confirmation window is only set when the minimum unconfirmed sequence number is reached. When acknowledged, the entire write chain slides, ensuring that any ACK out-of-order processing will not disrupt the consistent playback of the write sequence chain. Furthermore, monotonic indexing treats object storage as a topologically transparent sequential relay, eliminating inter-node PUT write conflicts and achieving multi-threaded safety without sacrificing concurrency.

[0083] Step 302: Call the Hucang time travel interface to check the latest uploaded version number interval. Perform incremental snapshot persistence and insert traceable markers , so that the edge view is consistent with the main lake view; Although the object storage has been sequenced, if the lake warehouse side does not solidify the snapshot in time, the cloud scheduler will not be able to locate the gap through differential scanning in step 4. Therefore, it is necessary to trigger the time travel interface immediately after each version number window is capped; so that the current version index Mapping back to the original write sequence number Indexed with timestamp interval; Then use the snapshot piecewise function Generate segmented metadata by sequence span to avoid metadata explosion caused by huge lists; then insert a traceable marker at the end of each segment , with Merkel root , version range and CRC32 check to ensure that a single segment can be independently verified; finally, when the cloud scheduler listens to the new traceable marker When an event occurs, the corresponding segment can be directly loaded for differential analysis without scanning the entire lake.

[0084] According to the write sequence span With version span Adaptive segmentation and a diagonal balancing formula ensure that the write sequence number and version number have equal impact, allowing the segment size to change smoothly with the growth rate of both, preventing exponential explosion at one end.

[0085] Among them: write sequence number span : The maximum sequence number minus the minimum sequence number in the capped window; version span : Version number difference within the same window; scale factor : , used to control the segment granularity; number of segments : The number of snapshot segments obtained.

[0086] Construct a backtracking marker at the end of each segment , whose contents are hashed Compression, generating formula:

[0087] Among them: Merkel root : The root fingerprint of the last block in the corresponding segment; the starting version : Segment starting version number; end version : Segment end version number; block : Cyclic redundancy check code of block payload; hash function : BLAKE3, consistent with the previous; backtracking mark : segment-level metadata anchor, 256 bits; The tag encapsulates the three-dimensional attributes of sequence, version and content at once. The cloud only needs to sign for the traceable tag. The segment-level consistency can be determined, which greatly reduces the reconciliation I / O. After the snapshot is persisted, the lake warehouse returns the confirmation vector To the edge, add the solidified version number set to the confirmation set :

[0088] Where: Confirmation vector , a set of version numbers that have been verified by Hucang; confirm the set : Defined in step 1, used for replay pointer , , indicating that the data or data block has been verified; Bidirectional write-back reports the main lake confirmation to the edge immediately, making the replay pointer A closed loop is also achieved at the version level to avoid repeated uploading or accidental deletion of unsolidified fragments.

[0089] The Hucang snapshot solidification process is divided into several segments. As the core of granularity control, by writing sequence number span With version span Diagonal coupling automatically finds a balance between metadata and snapshot size, so that incremental persistence is neither over-split nor bloated; and with the help of embedded traceable markers Merkel , version interval and CRC are hashed and encapsulated at once. Hucang only needs to verify a single segment fingerprint to confirm data integrity, which fundamentally avoids the expensive I / O of long chain directory comparison. Subsequently, Hucang will confirm the vector Write back to the edge and directly drive the replay pointer of step 1 Close the corresponding gaps to form a timely closed loop and prevent redundant and repeated uploading.

[0090] The replication rate constructed in step 301 -Flow-through confirmation window -Monotonic version index The ternary pipeline system can still write sequence numbers under strong fluctuation links. Push slices sequentially without blocking; and step 302 further uses snapshot segmentation function -Retraceable marking - A three-stage closed loop of bidirectional consistency write-back maps the sequential versions in the object storage into incremental snapshots that can be referenced by the lake warehouse time travel, and returns the confirmation results to the absorption layer.

[0091] The two steps are closely linked: the continuous version number interval generated by the former is directly input into the snapshot segmentation algorithm of the latter, and the confirmation vector written back by the latter The replication window is then converged in real time, forming a self-driven closed loop of push-solidification-confirmation. Compared to the traditional scheduled batch upload + offline snapshot model, this solution outputs a verifiable version number at the moment of upload and writes back the confirmation vector at the moment of snapshot solidification, reducing the cloud-edge end-to-end consistency latency from minutes to sub-seconds. At the same time, the traceback mark With Merkel Root Binding allows differential scanning to locate gaps by processing only segment-level fingerprints, providing a millisecond-level starting point for rapid reconstruction in step four.

[0092] In step 3, the edge node has used the replication rate , version index With traceable mark The slice-root pair is pushed to the object storage and solidified as an incremental snapshot; however, this process only guarantees that the write is successful, not that the content can be immediately read by the model. Before entering the cloud training pipeline, there are still two potential faults: First, although the snapshot is formed, its sequence chain is Gaps can occur due to link jitter, which can trigger I / O blocking during training if not discovered in time. Second, redundant slices have been written, but if the cloud does not rebuild the missing blocks and refresh the time travel index in the first place, , the subsequent samplers will have difficulty in sequentially replaying from the continuous address space, causing the gradient descent process to stall.

[0093] Step 4: The cloud-based training scheduler monitors segment-level marking events, first using segment hashing and the Bloom-skip list structure to quickly perform a differential scan on the write sequence chain. After locating the gap, it uses the slice location vector to align the minimum data slice set at once. The GPU inverse matrix is ​​then used to quickly reconstruct the missing blocks and graft and refresh the time travel index and the global Merkle tree. This allows gaps to be closed immediately upon discovery, ensuring continuous consistency of training data.

[0094] Step 401: Trigger the differential scanner in an event-driven manner , locate the write sequence chain in the shortest time The set of all gaps in , and outputs a missing segment tuple that can be directly called by the subsequent reconstruction module .

[0095] The frequency of incremental snapshot generation is higher than the cloud batch training cycle. If full table scans are still used to compare sequence chains, metadata scans will drift within the TB range due to the mixing of hot and cold areas. Therefore, it is necessary to use previously written traceable markers. As a logical anchor point, we first coarsely locate the possible gap segment, and then perform a fine-grained skip table scan within the segment; thus reducing the original O(N) full-scale comparison to a sparse comparison of O(logN+|gap|); Then, a hash is used to determine whether the segment is complete. If it is incomplete, it will jump to the sequence level; then the gap found will be immediately written into the gap set. And broadcast it to the reconstruction scheduler; finally, a three-stage pipeline is formed: segment fingerprint → sequence number jump table → gap set, so that the differential positioning can match the upload rate without dragging down the GPU standby time.

[0096] To quickly exclude most complete segments, the differential scanner first calculates a segment-level hash for each new segment. :

[0097] Including: traceable mark : The segment fingerprint inserted in the previous step, 256 bits. Merkle root at the end of the segment : The integrity fingerprint of the last block in the segment, 256 bits; hash function :BLAKE3, unchanged; segment-level hashing : A quick fingerprint for comparison with historical snapshots.

[0098] If the segment-level hash If the data is consistent with the inventory record in the lake warehouse, the segment can be declared complete and the sequence number level scan can be skipped directly. Otherwise, the next technical point is to proceed to the next technical point. Within the segment that requires fine-grained comparison, the differentiator establishes a secondary structure for the write sequence number chain:

[0099] in: :Indicates the sequence Apply Bloom filter to store the existence of sequence number with 0.01 false positive rate; Skip array :Indicates the use of sequence SkipList structure, with a first-level index for every 8 serial numbers, and O(1) jumps; Binary structure : Represents a composite value composed of two data structures, ensuring low memory and providing random access; In the sequence number dense segment, Bloom is used to filter the large segment existence area, and then Skip is used to accurately locate the missing number, and the gap is found with two memory accesses. After the missing number is found, the gap priority is constructed using the following function :

[0100] Among them: Serial number span : notch length, unit piece; time aging coefficient : The number of seconds between the gap and the current time; weight , :satisfy , default 0.6,0.4; gap priority : The larger the value, the earlier the reconstruction.

[0101] Comprehensively consider the gap length and time aging coefficient to avoid long-term suspension of small but old gaps and to prevent new large gaps from slowing down training. By marking the three delicate links of hash-jump table-priority, the PB-level snapshot difference is reduced to memory-level operation, providing an accurate and well-sorted gap set for step 402. .

[0102] The differential scanner uses a three-level pipeline of prior screening, jump table indexing, and gap priority to reduce the dimensionality of snapshot verification of PB-level lake warehouses from linear traversal to fingerprint positioning + sparse jump. First, segment-level hashing The system can exclude most complete segments by reading very little metadata, significantly reducing disk addressing and cache jitter; then, the binary jump table The memory resident overhead is locked to a few percent of the total number of sequence numbers, but the missing number can be accurately located within two memory accesses, breaking through the traditional limitation of Bloom filtering that can detect but not locate; finally, the gap priority The introduction of a time aging parameter tightly couples reconstruction scheduling with the training cycle, thus avoiding order starvation, such as insufficient capacity starvation and long-term stagnation of small gaps. The coordinated implementation of these three elements enables a sub-second closed loop between differential analysis, positioning, and dispatch, enabling cloud resources to complete differential analysis within idle GPU slots without blocking iterations.

[0103] Step 402: Using the gap set output by the differentiator , calling the fast rebuild engine Collect from redundant slices in minimal I / O Reorganize missing blocks and refresh the time travel index With Merkle Tree , ensuring continuous cloud-edge consistency.

[0104] The reconstruction process must be fast while ensuring that the snapshot version consistency is not destroyed. Simply relying on conventional RS decoding will result in multiple remote reads due to the wide distribution of fragments. Therefore, the fragment location vector is introduced. , aggregate enough fragments at a time; thus achieving the minimum number of copies and one-time alignment; and then using the GPU matrix core Parallel inverse operation, tens of milliseconds to solve the original block; then use the block-level root Then embed the global Merkle tree and update only the affected paths according to the forest-graft algorithm; finally write back the updated time travel index and segment markers , declaring the gap closed.

[0105] Generate patch location vector :

[0106] Where: slice positioning vector : A list of nodes that need to be reached in a remote call; node : Edge or cloud storage nodes storing slices; slices : Data or check pieces corresponding to missing blocks; minimum number of data pieces : defined by step 2, 4–16 slices; Executed on Tensor Cores:

[0107] Among them: inverse matrix : Corresponding redundancy ratio The Vandermonde inverse, dimension ; Slice vector : Positioning vector according to the image The smallest piece of data that can be aggregated.

[0108] Reconstruction Block : The original byte sequence of the missing block, representing the final block data. It is the result obtained by applying the inverse encoding function to a certain input.

[0109] GPU performs matrix-vector multiplication in about The refresh operation is done within 10 seconds, and the latency is much lower than that of CPU shard reorganization. The time travel index and Merkle tree are incrementally refreshed. The refresh operation follows: ,

[0110] Among them: key : Associated reconstruction block; index : Hucang time travel index, mapping write sequence number to object URI; grafting function : Recalculate node hashes only on the affected path in the Merkle tree; global tree : The top-level Merkle tree covering all blocks in the current training cycle; Incremental mode avoids recalculating the entire tree, ensuring that lake warehouse query response times are not slowed by rebuild operations, while also ensuring that indexes are immediately available for training and reading. By weaving slice positioning, GPU decoding, and metadata refresh into the shortest path of positioning-decoding-grafting, gap closure latency converges to low milliseconds, providing a seamless and continuous data surface for sample playback in step five.

[0111] Fast reconstruction engine using slice positioning vectors Pull together the minimum data set at one time, and then use GPU inverse matrix High concurrent decoding, then using the grafting function Only the Merkle tree nodes of the affected paths are recalculated, and the entire gap filling process hardly touches irrelevant cold data. This design can complete cross-node aggregation in a single I / O, avoiding the multiple rounds of pull required for conventional RS decoding; GPU computing can The chip decoding delay is reduced to hundreds of microseconds, which is much lower than the CPU-level millisecond scale; more importantly, incremental grafting allows the global tree Maintaining a stable root without causing the entire tree to be rehashed due to partial reconstruction, thus ensuring time travel indexing It is immediately readable after completion.

[0112] Step 4 uses the hash of the tag from step 401 -Bloom-Skip jump table -Priority function Three differential links are used to locate the write sequence number gap at high speed in the ever-changing cloud side lake warehouse; then step 402 uses the slice positioning vector —GPU inverse matrix -Graft Refresh The three-stage reconstruction pipeline fills in the gaps and updates the index and Merkle tree in real time. The two steps are closely linked and complement each other.

[0113] Compared to traditional offline verification and batch reconstruction, this method deeply combines segment tags with sequence number indexing, first performing sparse positioning at the logic layer and then batch reorganization at the data layer, significantly shortening the differential-reconstruction closed loop. Furthermore, through a combination of GPU decoding and incremental grafting, high-throughput matrix calculations and low-overhead metadata updates are performed in parallel, avoiding the gap caused by reconstruction being unable to query immediately. This brings the cloud-edge view into a truly continuous and consistent state: any new snapshot is verified and completed within sub-seconds, allowing the training scheduler to safely pull samples without explicit waiting.

[0114] In step 4, the cloud training scheduler has ensured the time travel index of the lake warehouse through differential scanning and fast reconstruction With the global Merkle tree No gaps; however, this only completes the data plane continuity.

[0115] Once the data is read by the training process, it can still introduce fine-grained corruption in the sample-load-forward chain due to network rollovers, disk overwrites, or GPU memory soft errors. Traditional deep learning frameworks often skip read failures or fill anomalous samples with zeros. This can cause gradient oscillations at best and introduce hidden biases at worst. This is especially true in self-supervised or contrastive learning scenarios, which rely more on sample integrity than conventional classification tasks.

[0116] Step 5: During the training sampling phase, GPU-CPU collaboration is used to generate sample Merkle proofs for immediate integrity verification. Any mismatched samples are written to the damage queue and replaced with tensors generated in the same batch through neighbor prediction consistency interpolation. Gradient stability gating monitors disturbances in real time and automatically adjusts weights. Repair events are written to the repair log chain and the repair log root is updated, thereby achieving online self-healing and auditable traces.

[0117] Step 501: In the training sampler Read each sample When , the Merkle proof is generated synchronously And perform integrity verification; if the verification fails, it is immediately determined to be a damaged sample and triggers the upstream repair pipeline.

[0118] During GPU training, I / O bandwidth and computing power often show a scissors gap. If synchronous verification blocks reading, the throughput will be reduced to the single-thread level. Therefore, hash path inference and GPU decoding pipeline need to be run in parallel, and hash calculations can be performed using TensorCore idle cycles. This allows for real-time verification without reducing GPU utilization. This is achieved through segmented hash path buffering. Batch splice multiple certificates to reduce PCIe round trips; then mark them as damaged when verification fails Add the sample to the repair queue And carry the write serial number Finally, the main training thread continues to advance without any awareness, while the background repair thread takes over the interpolation logic.

[0119] Three lines of description - To reduce PCIe round trips, hash path calculations are split into leaf hashes within the GPU layer With CPU layer parent hash Two levels: ,

[0120] Among them: sample data : Normalized Tensor, stored in GPU; write sequence chain : Generated in step 1, 64bit.

[0121] Timestamp Index : 64bit; leaf hash : GPU calculation results, 256 bits; brother hash : Buffer from segmented hash paths Retrieve; Parent Hash : CPU-side synthesis, pushing up the Merkel path.

[0122] After the GPU completes the high-throughput BLAKE3 leaf hash, it returns to the deep pipeline, and the CPU only splices the previous layer, using NVLinkP2P to minimize cross-bus data transfer. The complete proof is split into fixed 1KB segments and placed in the ring buffer Provides batch pull for the GPU to resolve DMA fragmentation caused by random paths of varying lengths.

[0123] Buffer capacity configuration:

[0124] Where: batch size : Current GPU mini-batch size; multiplication factor : , to ensure that the CPU can fill the buffer before it overflows; ring buffer : Manage parent hash shards.

[0125] The ring design rotated by batch can ensure lock-free concurrency of buffers under large batch reading. The CPU only needs to fill the slices, and the GPU DMA takes all the required paths at once. Still doesn't match the global root, generating:

[0126] And write:

[0127] Where: Error description : Record the first layer mismatch hash; damage mark : Single sample integrity failure event; queue to be repaired : FIFO capacity 10K, used for asynchronous interpolation thread.

[0128] Using a queue, samples that failed synchronization verification are transferred to the backend. The main training thread then simply sets the damaged sample mask to 0 and continues gradient accumulation. Step 501 utilizes GPU-CPU collaborative hashing, circular path buffering, and damaged queue isolation to implement a lightweight pipeline with parallel verification and managed exceptions, creating a time window for subsequent interpolation.

[0129] The cost of sample-level integrity verification is almost equivalent to that of normal tensor loading: leaf hashing is done on-chip when the TensorCore is idle, without occupying the main computing power; parent hashing can be converged by splicing in the CPUL3 cache without touching the remote DRAM; the ring buffer cuts the variable-length Merkle path into fixed-length slices, completely eliminating DMA fragmentation. At the same time, any verification failure is marked as damaged. Capture instantly and push into the repair queue , the main thread only needs to set the mask to zero to continue gradient accumulation, and the damaged samples are dragged out of the main batch, minimizing the perturbation of the training distribution and gaining a millisecond-level window for subsequent interpolation.

[0130] Step 502: Queue to be repaired Each damaged sample in Perform forecast consistency estimation , online interpolation generates alternative tensors And update the repair log synchronously , ensuring that the gradient flow is continuous and the audit chain is closed.

[0131] Directly discarding bad samples will cause fluctuations in the number of valid samples in the batch, causing BN / LayerNorm statistical drift; therefore, it is necessary to generate alternative tensors within the same training iteration; interpolate based on the nearest time window and the current EMA of the model; and then write the repair event to the log Merkle chain after the interpolation is completed. and fix the log root Backfill segment markers; then ensure that during audits, the same Merkle path can be used to verify that the sample has been repaired and the repair strategy is transparent; ultimately, the gradient is stabilized without introducing black box changes.

[0132] The neighbor window prediction consistency estimation algorithm formula is as follows:

[0133] Among them: neighbor set : Same mini-batch with alternative tensors The 4 most recent samples in the time series; alternative tensor : Used for forward and gradient backpropagation; Weight : , the sum after normalization is 1; the normalization factor : ; Exponential moving mean : The mean forward output of the model; interpolation balance coefficient : , combat overfitting; The neighbor weights are inversely related to the sample-mean distance, ensuring that the replacement tensor does not deviate from the distribution; interpolation balance coefficients Then global smoothing is injected to reduce the noise of small data sets. After interpolation, the perturbation of the gradient distribution needs to be evaluated and the gradient perturbation is defined as :

[0134] like , then increase the interpolation balance coefficient and recalculate; Where: current gradient : Backpropagation gradient with replacement tensor; baseline gradient : Gradient snapshot before replacement; stable threshold : ; Gradient perturbation : relative rate of change; Among them, dynamic gating ensures that the interpolation strategy does not cause 15% gradient explosion and keeps the training stable. Generate repair entries for each interpolation :

[0135] And append the repair log :

[0136] Where: Timestamp : Fix completion nanosecond timestamp; fix entry : Single event fingerprint; repair log : Collect all repair events in sequence; repair log root : Log Merkle tree root, 256 bits; Will fix the log root Writing back to the segment marker allows the auditor to verify the repair record and replacement value with only the root, forming a data-repair-proof trinity chain.

[0137] Step 502 achieves a full closed loop of repairing damage and verifying repairs through predictive consistency interpolation, gradient stability gating, and repair log chain, and closes the loop at both the parameter layer and the audit layer.

[0138] Generate in-distribution surrogate tensors using batch-intra-neighbors and model sliding mean , ensuring that the number of valid samples in the same iteration remains unchanged, and the BatchNorm mean and variance do not need to be re-estimated; the gated gradient perturbation Real-time detection of the impact of interpolation on the gradient, if the threshold is exceeded, the interpolation balance coefficient will be adaptively converged immediately , to avoid excessive repair and gradient explosion; each interpolation is repaired Write Merkelized logs to generate repair log roots And backfill the segment markers, allowing auditors to verify the repair details based on the root alone.

[0139] On the one hand, GPUHash calculations leverage spare TensorCores to reduce integrity verification costs to microseconds. On the other hand, dynamic feedback from interpolation-gradient gating ensures that the perturbation of the gradient flow by repair samples does not exceed the threshold, maintaining optimizer stability. Furthermore, the repair log chain embeds the repair and trace into the same Merkle domain, providing a self-consistent and irrefutable event sequence for the evolutionary freeze in step 6.

[0140] With the sequential implementation of the first five steps, a full closed-loop process has been completed, from edge acquisition, redundancy correction and erasure, snapshot storage, differential reconstruction in the cloud, to online repair during training. However, the completion of the training job does not necessarily mark the end of the data lifecycle. Three unresolved follow-up requirements remain: First, regulatory compliance—the snapshot versions, repair logs, and operation events accumulated during training must be sealed as an immutable history for post-audit or model accountability. Second, resource controllability—with the termination of training, the redundant slices overprovisioned during the encoding phase have lost their immediate recovery value. If not cleaned up in a timely manner, they will overwhelm the capacity budget of NVMe cache and object storage. Third, traceable freezing—the snapshot freeze must ensure that the three chains of data, repair, and history are atomically locked under the same fingerprint, otherwise subsequent cleanup will easily cause reference chain breaks.

[0141] Step 6: Generate a history list within the write-quiet window after training. This list is combined with the snapshot root and the repair log root through a sealing function to form a frozen root to lock the data. Then, redundant checksums are deleted based on the retention probability function and erased smoothly using write-amplification negative feedback. After completion, a cleanup root is generated in the revision table, which is then combined with the frozen root to form the final audit root, completing a one-stop loop for archiving, cost recovery, and traceable auditing. Step 601: Generate history records, atomically freeze the current snapshot state and lock the audit chain root, so that the snapshot content fingerprint , Repair log root and historical issuance roots The three parties converge to a single frozen root , providing an irrefutable reference for subsequent cleanup.

[0142] The end of training is often accompanied by a high-frequency mix of write and read operations. If frozen directly, it will compete with the last batch of gradient writes for metadata locks; therefore, a freeze scheduler is needed. After listening to the training completion event, enter the write silent window; thus ensuring that all cache writes are implemented; and then based on the version index Calculate the freeze vector And lock the write permission; then repair the log root With snapshot root Embed the history list in sequence and calculate the history issuance root ;Finally use the on-chain sealing function Synthetic frozen roots , and broadcast it to both the cloud and the edge.

[0143] Freeze vector generation and write silent window, in which, in order to evaluate whether the freezing conditions are met, the freeze vector is introduced :

[0144] Where: Active Write Latency : The time difference between the last write operation in the past 30 seconds and the current time, in seconds; value .

[0145] Buffer residual depth : Write buffer bytes not yet flushed to disk; refresh completion rate : Persisted data / data to be persisted, scope ; Freeze vector : Evaluate whether the write silent window meets the threshold; If both meet 、 、 That triggers freezing, where: silence threshold : Buffer threshold : .

[0146] Among them, the three-dimensional indicators jointly prevent the lock from being grabbed and frozen when the tail batch gradient impact is still being refreshed, ensuring data consistency. After the freezing condition is met, the scheduler generates evolution entries in the order of write sequence numbers. :

[0147] And compiled into a historical list : ,

[0148] Where: training time scale : The nanosecond time when the sample completed the last gradient update; evolution entry : Single fingerprint; History List : Complete version sequence at the time of freezing; evolutionary issuance root : Merkle root of the evolutionary list; Process the slice (such as hashing or encoding) to produce a root value ; Thus, the three-dimensional information of version, time and serial number is fixed once and for all into an unalterable list, providing hard evidence for investment accountability. Final Frozen Root Synthesis:

[0149] Where: Snapshot root : Global root of the last time travel snapshot; fix log root :Step 5 output; Random Salt : 128bit, session random number anti-collision; frozen root : Final audit fingerprint.

[0150] By merging the three roots with salted hashing, any change in a single root will cause a frozen root mismatch, and the salt value prevents cross-project fingerprint reuse attacks. Step 601 integrates write silent detection, evolution list, and chain sealing to achieve a continuous chain of moment capture-entry sealing-fingerprint locking, providing a solid baseline for asynchronous cleanup.

[0151] The evolutionary freeze scheduler freezes vectors within the write-quiet window Accurately capture the I / O convergence moment, and then use a single chain sealing function to restore the snapshot to the root , Repair log root and historical issuance roots Aggregate into a unique frozen fingerprint This three-in-one strategy inseparably locks data, repairs, and historical operations, completely avoiding the governance vacuum in the traditional archiving process where snapshots and operation logs belong to two separate chains and are difficult to align later. At the same time, the freeze action is only triggered under silent conditions and does not compete with the tail batch gradient write for file handles, eliminating the risk of metadata deadlocks common during the archiving period.

[0152] Step 602: Freeze the root After release, the scheduling background executes the three-stage process of redundant slice recovery-cache release-version table revision to retain the threshold Free up space for standards efficiently and lock the audit chain.

[0153] After freezing, more than 90% of redundant slices are no longer valuable for online training; therefore, a redundant recovery strategy is used Only keep cross-node hash slices A subset of ; thus significantly compressing object storage bills; and then through cache recycling scheduling vector Control the NVMe asynchronous deletion rate to avoid wear peaks; then revise the object storage version table after the slice deletion is completed And change the entry to clean up the root The frozen segment is added in a formal manner to ensure log self-consistency; eventually the system capacity returns to the baseline cost, and the audit chain can still be verified by a single root.

[0154] Execute redundancy recovery strategy , where the retention probability is defined as :

[0155] Among them: minimum number of data pieces : Step 2 defines the recoverable threshold; slice index : Sequence number in the slice set; retention threshold : , the higher the value, the more aggressive the cleanup; the retention probability :piece The opportunity to be preserved.

[0156] Make sure at least Data slices must be retained, and check slices are deleted with an exponential decay probability; this strategy is both error-tolerant and space-efficient. To control the NVMe lifecycle, the recycling scheduler uses:

[0157] Adjust the erase and write rates and dynamically limit the speed to prevent concentrated cleaning from triggering SSD write amplification, thereby extending the media life:

[0158] in: Indicates time The I / O write rate under the device. This rate varies with the wear of the device; Basic Rate : Factory recommended IOPS; daily write volume : Today's erase and write volume, GB; upper limit of write volume : Daily health limit; wear coefficient : , negative feedback intensity; erasure depth : The number of blocks touched by a single Trim.

[0159] After the slice is deleted, the object storage side updates the version record:

[0160] , represents the updated version index set, which contains all the indexes retained after the deletion operation; , represents the original version index set, which contains the indexes of all initial versions; , represents the deleted index set, including the index of the deleted version; , represents the set difference operator; And calculate:

[0161] , which means that the updated version index collection The value obtained by performing the hash operation; , represents a hash function, which is used to encrypt or compress input data (such as a version index set) to generate an output value of a fixed length; :Indicates The indexes in the collection are sorted. After sorting, they are input into the hash function for hash calculation to generate the corresponding hash value; Finally:

[0162] Write frozen segments + clean up patch joint roots; in, Indicates the frozen version hash value, Indicates the hash value obtained in the second formula, representing the updated version index, key : Base64 serial number-timestamp string; clean random salt :128bit.

[0163] Audit the final root : The only verification fingerprint after sealing + cleaning.

[0164] Cleanup actions are incorporated into the audit chain using a joint root approach to ensure that deletion operations can be reproduced historically. Step 602 First, redundancy is pruned using a probability retention strategy. Then, write amplification and negative feedback are used to smooth out writes. Finally, the cleanup root is stamped at the version table level, achieving a post-processing closed loop that verifies deletion.

[0165] The frozen vector of step 601 - History List -Sealing function The three-stage structure accurately captures the silent window at the end of training and saves the snapshot , Repair log root and historical issuance roots Compressed to a single frozen root , from this data view into the read-only-auditable state. Then step 602 uses the redundancy recovery strategy -Cache eviction vector - Clean up the roots The three-stage flow safely removes the over-allocated verification piece and links the result fingerprint with the frozen root to form the audit final root The frozen root output by the former serves as the input for the cleaned fingerprint, while the audit final root generated by the latter becomes the final node in the chain of evolutionary records, ensuring that the original can still be verified after cleansing.

[0166] The recovery phase uses an exponentially decaying redundancy recovery strategy Only the smallest recoverable slice is retained and very few check slices, which not only maintains the bottom line of disaster recovery, but also immediately reduces the object storage bill to a single-digit percentage of the active period; at the same time, the cache recycling scheduling vector The trim rate is smoothed based on real-time write volume feedback, which reduces the long tail of SSD daily writes and prolongs the media life without sacrificing cleaning speed. Merge again with the frozen root to form the final audit root External supervision only needs to check this one to verify what was stored at the time and what was deleted afterwards, thus realizing a closed loop where deletion can also be verified.

[0167] Different from the common backup and deletion or periodic archiving + manual issuance in the industry, the freeze-clean-conjoined root mode of this solution combines data retention, redundant recovery and traceability audit into a one-time atomic operation: on the one hand, the retention probability Accurately control costs to reduce object storage overhead over the model lifecycle; on the other hand, write amplification negative feedback Extend the life of SSD and avoid the media avalanche caused by cold storage → hot deletion; more importantly, the audit will ultimately Any future review can verify what was stored, what was deleted, and when it was deleted with just a single root. From the moment of its birth to the final audit Until it is permanently sealed, the data has completed a self-consistent, trustworthy, economical and verifiable closed loop, laying a clean and sustainable storage foundation for the next round of learning models.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a division of some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0172] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data storage method for an artificial intelligence learning model, characterized in that: include, At the edge high-speed absorption layer, multi-source data streams are assigned monotonic write sequence numbers and merged into sample blocks in real time, providing reliable timing for subsequent erasure coding and reconciliation, and ensuring fault replayability. Perform asynchronous erasure coding on the sample block based on the write sequence number to generate cross-node redundant slices, calculate the Merkle root and write it into the local cache for integrity verification and differential scanning; The edge side pushes the slices and Merkle roots to the object storage in the order of the write sequence through the bandwidth-aware replication pipeline, and calls the time travel interface to solidify the incremental snapshot and record the backtracking mark; The cloud training scheduler monitors snapshot events to trigger differential scanning, compares the write sequence number chain, and if a gap is detected, reconstructs the missing segment using the redundant slices and refreshes the time travel index to maintain consistency; When replaying samples during the training process, a Merkle proof is generated for each sample to verify integrity. If the verification fails, the damaged sample is marked and repaired using prediction consistency interpolation, and the repair event is recorded at the same time. After training is completed, the snapshot is frozen in the write silent window to generate a history record and lock the frozen root. Then, redundant slices are asynchronously deleted to release cache space, and the object storage version table is revised to generate the final audit root.

2. The data storage method for artificial intelligence learning mode according to claim 1, characterized in that: The write sequence number chain is generated by a two-stage write sequence number function determined by the logic clock frequency and the carry window size. The write sequence number remains globally monotonically increasing in a multi-threaded environment and is prevented from wrapping around by window folding before the count overflows. It is bound to the block merge cache to guide the windowed splicing of multi-source perception streams. The block merge cache has a limited capacity and is cyclically dequeued in the order of the write sequence numbers.

3. The data storage method for artificial intelligence learning mode according to claim 2, characterized in that: After generating the sample block, calling the asynchronous erasure coding queuer, the queuer calculates the priority according to the arrival interval of the merged block and the historical missing block rate, writes the priority into a preemptive priority queue, and inputs the block with the highest priority into the erasure encoder; Timestamp indexes and Merkle root summaries are appended at the slice layer and written to the local cache in a key-value manner to ensure atomic consistency of blocks and metadata. The key-value pairs are stored in a reserved transaction area of ​​a high-speed solid-state cache.

4. The data storage method for artificial intelligence learning mode according to claim 3, characterized in that: Asynchronous erasure coding uses a Vandermonde matrix with an adjustable redundancy ratio, dynamically adjusts the number of data slices and the number of check slices, and uses a bandwidth-aware slice distribution algorithm to distribute the generated slices to multiple nodes with exponential probability; The coding queuer uses arrival interval, missing block rate and link margin as parameters for real-time sorting to ensure that data slices are written to high-bandwidth nodes first and check slices are elastically distributed with the load. The multiple nodes include edge nodes and central nodes.

5. The data storage method for artificial intelligence learning mode according to claim 4, characterized in that: After erasure coding is completed, the block-level root is calculated through the hierarchical Merkle tree and the sub-root level index is generated. The write sequence number and timestamp index are concatenated as the key value to write the slice and the Merkle root atom into the local solid-state cache; The key value is a unique serial number timestamp string for subsequent differential scanning and lake warehouse time travel positioning. The solid-state cache uses non-volatile storage and supports direct key addressing, and provides a block root index for subsequent differential scanning.

6. The data storage method for artificial intelligence learning mode according to claim 5, characterized in that: The replication pipeline uses an adaptive rate control law to adjust the sending rate in real time according to link utilization, buffer depth and available bandwidth, and the pipeline confirmation window limits the confirmation order based on the maximum number of flying chips; Any out-of-order confirmation will only slide as a whole after the smallest unconfirmed sequence number is acknowledged, ensuring that the push order is strictly consistent with the write sequence number and preventing reverse overwriting.

7. The data storage method for artificial intelligence learning mode according to claim 6, characterized in that: When calling the Hucang time travel interface to solidify the incremental snapshot, the number of snapshot segments is determined by the square root balance formula based on the write sequence span and version span; A traceable marker containing the block root, version boundary and cyclic redundancy check code is inserted at the end of each segment and a segment-level hash is generated so that differential scanning can directly compare the marker to determine the integrity of the segment without the need for directory traversal. The segment-level hash result is written into the incremental snapshot metadata area.

8. The data storage method for artificial intelligence learning mode according to claim 7, characterized in that: The differential scanner first calculates the segment-level hash value for the traceable marker and the end-of-segment Merkle root. If the hash does not match, it builds a binary index structure consisting of a Bloom filter and a jump table with eight sequence numbers to locate the write sequence number gap within two memory accesses. The reconstruction priority queue is calculated based on the gap length and the time aging coefficient for the reconstruction engine to call. The gap priority queue is updated in real time in the memory.

9. The data storage method for artificial intelligence learning mode according to claim 8, characterized in that: Fast reconstruction uses the slice location vector to pull the minimum set of data slices that meet the recovery conditions at one time, schedules the Tensor Core to perform inverse Vandermonde matrix operations to restore the missing blocks, and uses the incremental grafting algorithm to recalculate only the affected path nodes of the global Merkle tree; At the same time, appending new block key values ​​to the time travel index keeps queries consistent with the snapshot, which involves the row consistency of the time travel index.

10. The data storage method for artificial intelligence learning mode according to claim 9, characterized in that: For online verification, the CPU splices the parent hash after the leaf hash is calculated by the tensor core, and the ring buffer manages the proof path with a fixed length of one kilobyte. If the verification fails, a damage mark containing the write sequence number, timestamp, and first-layer mismatch hash is generated and written to the queue to be repaired. The main training thread ignores damaged samples in batches through masks to maintain the tensor shape, and the damage mark is stored in the shared memory segment of the lock-free ring buffer.

11. The data storage method for artificial intelligence learning mode according to claim 10, characterized in that: The interpolation repair algorithm selects the nearest neighbor samples in the same batch and combines them with the model sliding mean to generate a replacement tensor, and monitors the gradient perturbation rate in real time. If it exceeds the preset stability threshold, it automatically increases the mean weight and re-interpolates, and writes the repair entry containing the write sequence number, replacement tensor summary and gradient perturbation into the repair log and updates the repair log root. The repair log root is synchronized to the upper node of the global Merkle tree.

12. The data storage method for artificial intelligence learning mode according to claim 11, characterized in that: A freeze vector is formed by monitoring three indicators: active write latency, buffer residual depth, and refresh completion rate. When the silent threshold and buffer threshold are met and the refresh completion rate is 100%, a history list sorted by write sequence is generated to calculate the history issuance root. The snapshot root, repair log root, and history issuance root are salted and hashed to generate a freeze root, and the write permission is broadcast and locked. The history list is securely written to the immutable object storage bucket.

13. The data storage method for artificial intelligence learning mode according to claim 12, characterized in that: After generating the frozen root, the system retains at least the minimum number of data slices according to the retention probability function and performs exponential decay deletion on the parity slices. During the deletion process, the erase rate is controlled by write amplification negative feedback scheduling to extend the life of the solid-state media. After completion, the object storage version table is revised to generate a cleanup root, which is hashed together with the frozen root to obtain the audit final root, and the audit final root is written into the metadata index of the frozen segment.

Images