Data storage method for artificial intelligence learning mode

Abstract

The application discloses a data storage method for an artificial intelligence learning mode and relates to the technical field of computer data storage. Step one: multiple source perception streams are merged into blocks in real time through monotonic writing of serial numbers at the edge to provide replayable timing; step two: asynchronous erasure coding is performed on the blocks, and a Merkle root is calculated and written into a local cache to realize double protection of fault tolerance and integrity; step three: according to the network, slices and roots are sequentially pushed to object storage, and a time travel interface is called to solidify an incremental snapshot; step four: a cloud end listens to snapshot hash events, differentially scans a serial number chain, reconstructs a gap by using a slice, and refreshes an index to maintain consistency; step five: a training process generates a Merkle proof online to check a sample, damaged data is immediately interpolated and repaired, and an audit chain is recorded; and step six: after the training is completed, a change list and a frozen root are generated, redundant slices are cleaned up asynchronously, a version table is updated, and finally, a single-fingerprint traceable cost archive is formed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data storage, in particular to a data storage method for an artificial intelligence learning mode. BACKGROUND

[0002] With the wide deployment of edge computing nodes, the generation mode of training data evolves from single-point upload to multi-source perception stream parallel influx. Existing systems generally use near-real-time archiving mode based on device timestamps or file naming rules: heterogeneous terminals write raw samples to local files, then upload them to central object storage by batch scheduling tasks, and then complete static erasure coding and hash verification through offline scripts. To avoid frequent small file write amplification, the typical approach is to aggregate several files into an archive package by caching at the edge for a certain period of time, and then push the entire package. The cloud performs directory traversal and verification scripts before training, and only after verification passes does it start decompression and reading. Such a pipeline can meet the basic data integrity requirements in an environment with sufficient bandwidth and stable nodes; however, once the terminal goes offline, the link drops, or the node restarts, the upload order is often disordered, and manual log file comparison is required to find the missing segments, which seriously delays the training pipeline. More problematic is that, in order to ensure lossy recovery, existing systems often configure a fixed redundancy ratio for all archive packages and store complete verification pieces for a long time, resulting in a continuous increase in object storage costs and NVMe write amplification, especially in deep learning scenarios with frequent continuous iterations and A / B testing.

[0003] In the distributed artificial intelligence learning mode, how to establish a nanosecond-level sequential coordinate for multi-source perception stream on the edge side, and under the conditions of continuous fluctuations in cloud-edge links and random failures of nodes, to achieve continuous consistency and single-fingerprint auditability from data collection, redundancy coding, snapshot solidification, differential reconstruction, online repair during training period, to redundancy recovery after training, has become a core problem faced by the industry.

[0004] The existing batch archiving and offline verification-based solution lacks write sequence number chain control and real-time Merkle verification. Once the edge node restarts or the link fluctuates, causing a sequence number gap, the cloud can only locate the missing segment by full-scan or manual script comparison, with a delay of minutes to hours; if the training process starts during this period, it will read the broken sample stream, causing gradient explosion or model divergence. On the other hand, the fixed redundancy ratio causes the number of verification pieces and the data volume to increase linearly, and when the training version is iterated frequently, the object storage cost and SSD erase count increase sharply, which cannot meet the cost-controllable engineering indicators, nor can it meet the regulatory requirements for post-deletion evidence.

[0005] Therefore, there is an urgent need for a unified ontology mechanism that can adapt to bandwidth, dynamically adjust redundancy, verify completeness in real time, and automatically freeze and recycle redundant resources after training ends, otherwise it will be difficult to balance timeliness, reliability, economy and traceability in large-scale deployment. SUMMARY

[0006] (I) Technical problems solved

[0007] In view of the deficiencies of the prior art, the present application provides a data storage method for artificial intelligence learning mode, which merges multiple source perception streams into blocks in real time through monotonic write sequence numbers at the edge to provide replayable timing; performs asynchronous erasure coding on the blocks and calculates the Merkle root to write to the local cache, realizing double protection of fault tolerance and integrity; pushes the slices and roots to the object storage according to the network in order and calls the time travel interface to solidify the incremental snapshot; the cloud listens to the snapshot hash event, differentially scans the write sequence number chain, uses the slices to rebuild the gap and refreshes the index to maintain consistency, and finally forms a single fingerprint traceable cost archive; thereby solving the technical problems described in the background art.

[0008] (II) Technical solutions

[0009] To achieve the above purpose, the present application is implemented by the following technical solutions:

[0010] The data storage method for artificial intelligence learning mode comprises: assigning monotonic write sequence numbers to multiple source data streams at the edge high-speed absorption layer and merging them into sample blocks in real time, providing reliable timing for subsequent erasure coding and reconciliation comparison and ensuring fault replayability;

[0011] Based on the write sequence numbers, asynchronous erasure coding is performed on the sample blocks to generate cross-node redundant slices, and the Merkle root is calculated and written to the local cache for integrity verification and differential scanning;

[0012] The edge side pushes the slices and Merkle root to the object storage in order according to the write sequence numbers through a bandwidth-aware replication pipeline, and calls the time travel interface to solidify the incremental snapshot and record the backtracking mark;

[0013] The cloud training scheduler listens to the snapshot event to trigger differential scanning, compares the write sequence number chain, and if a gap is detected, uses the redundant slices to rebuild the missing section and refreshes the time travel index to maintain consistency;

[0014] When playing back the training process samples, a Merkle proof is generated for each sample to verify integrity, and if the verification fails, the damaged sample is marked and repaired using predictive consistency interpolation, and the repair event is recorded;

[0015] After training, a write-silence window is used to freeze the snapshot generation and lock the frozen root, and then the redundant slices are asynchronously deleted to release the cache space, and the audit final root is generated by revising the object storage version table.

[0016] Further, the write sequence number chain is generated by a two-segment write sequence number function determined by the logical clock frequency and the carry window size, and the write sequence number is globally monotonically increasing in a multi-threaded environment and avoids backtracking by window folding before the count overflows.

[0017] And it is bound to the block merge cache to guide the windowed splicing of multi-source perception flow, and the capacity of the block merge cache is limited and dequeued in the order of the write sequence number.

[0018] Further, after generating the sample block, an asynchronous erasure coding queue is called, which calculates the priority according to the merge block arrival interval and the historical block loss rate and writes it into a preemptible priority queue, and the block with the highest priority is input into the erasure coder.

[0019] The timestamp index is attached to the slice layer and merged with the Merkle root digest and written into the local cache in a key-value manner to ensure atomic consistency of blocks and metadata, and the key-value pair is stored in the reserved transaction area of the high-speed solid-state cache.

[0020] Further, the asynchronous erasure coding uses a Vandermonde matrix with adjustable redundancy ratio, dynamically adjusts the number of data slices and the number of check slices, and uses a bandwidth-aware slice distribution algorithm to distribute the generated slices to multiple nodes with exponential probability.

[0021] The encoding queue is real-time sorted with arrival interval, block loss rate and link margin as parameters, ensuring that data slices are written to high-bandwidth nodes first and check slices are distributed flexibly according to load, and the multiple nodes include edge nodes and center nodes.

[0022] Further, after the erasure coding is completed, the block-level root is calculated by the hierarchical Merkle tree and the sub-root level index is generated, and the write sequence number and timestamp index are spliced and written as key-value to the slice and Merkle root atomically written to the local solid-state cache.

[0023] The key-value is a unique sequence timestamp string for subsequent differential scanning and lake warehouse time travel positioning, the solid-state cache uses non-volatile storage and supports direct key addressing, and provides block root index for subsequent differential scanning.

[0024] Further, the replication pipeline adjusts the sending rate in real time according to the link utilization rate, buffer depth and available bandwidth through an adaptive rate control law, and the pipelined acknowledgment window limits the acknowledgment order to the maximum number of in-flight slices.

[0025] Any out-of-order acknowledgment is only slid as a whole after the minimum unacknowledged sequence number is acknowledged, ensuring that the push order is strictly consistent with the write sequence number and eliminating reverse coverage.

[0026] Further, the number of snapshot segments is determined by a square root balancing formula according to the write sequence span and version span when invoking the lake warehouse time travel interface to solidify the incremental snapshot;

[0027] A rollbackable marker containing the block root, version boundary and cyclic redundancy check code is inserted at the end of each segment and a segment-level hash is generated, so that the integrity of the segment can be determined by directly comparing the markers without directory traversal, and the segment-level hash result is written to the incremental snapshot metadata area.

[0028] Further, the differential scanner first calculates the segment-level hash value of the rollbackable marker and the end of the Merkel root, and if the hash does not match, a binary index structure containing a Bloom filter and an eight-sequence skip list is established to locate the write sequence gap within two memory accesses;

[0029] And according to the gap length and time aging coefficient, a reconstruction priority queue is calculated for the reconstruction engine to call, and the gap priority queue is updated in real time in memory.

[0030] Further, the fast reconstruction uses a slice positioning vector to pull a set of minimum data slices that meet the recovery conditions at one time, schedules a tensor core to perform inverse Vandermonde matrix operation to recover the missing block, and only recalculates the affected path nodes of the global Merkel tree through an incremental grafting algorithm;

[0031] At the same time, the new block key value is appended to the time travel index to maintain query continuity and consistency with the snapshot, and the query continuity involves row consistency of the time travel index.

[0032] Further, the online verification splices the parent hash by the central processing unit after the leaf hash is calculated by the tensor core, and the ring buffer manages the proof path with a fixed one-kilobyte slice length; if the verification fails, a damage marker containing the write sequence number, timestamp and first-layer mismatched hash is generated and written to the repair queue, the main training thread ignores the damaged samples by masking and maintains the tensor shape in batches, and the damage marker is stored in the lock-free ring buffer shared memory segment.

[0033] Further, the interpolation repair algorithm selects the temporally nearest neighbor sample and the model sliding mean in the same batch to generate a replacement tensor, and monitors the gradient disturbance rate in real time; if the preset stability threshold is exceeded, the mean weight is automatically increased and the interpolation is re-performed, and a repair entry containing the write sequence number, replacement tensor digest and gradient disturbance is written to the repair log and the repair log root is updated, and the repair log root is synchronized to the upper nodes of the global Merkel tree.

[0034] Further, by monitoring the three indicators of active write delay, buffer residual depth and refresh completion rate, a freeze vector is formed, when the silence threshold and the buffer threshold are met and the refresh completion rate is 100%, the evolution list sorted by write sequence number is generated to calculate the evolution signing root, and the snapshot root, repair log root and evolution signing root are salted and hashed to generate the freeze root and broadcast the lock write permission, and the evolution list is safely written into the immutable object storage bucket.

[0035] Further, after generating the freeze root, data pieces are retained according to a retention probability function, and exponential decay deletion is performed on the check pieces, and during the deletion process, the erase rate is controlled by write amplification negative feedback scheduling to prolong the service life of the solid-state medium.

[0036] After completion, the object storage version table is revised to generate a cleaning root, and the audit final root is obtained by jointly hashing the cleaning root and the freeze root, and the audit final root is written into the metadata index of the freeze segment.

[0037] (Three) beneficial effects

[0038] The application provides a data storage method for artificial intelligence learning mode, which has the following beneficial effects:

[0039] Based on the write sequence number chain and block merging cache, a nanosecond-level unambiguous time sequence coordinate is established in place on the edge side, so that the multi-source perception stream can be accurately replayed in any node failure, link jitter or device restart condition, and the sorting cost is migrated from the cloud to the collection end, directly promoting the subsequent asynchronous erasure coding into a pure parallel matrix operation path, significantly shortening the write path and reducing the sorting-checking double overhead.

[0040] The asynchronous erasure coding atomically binds the adjustable Vandermonde matrix, bandwidth-aware slice distribution and hierarchical Merkle tree, and the coupling of redundant slices and integrity fingerprints is completed when each block is generated, the redundancy ratio is adjusted in real time according to the link margin without pausing writing, and hierarchical hashing allows local rewriting to only recalculate two layers of nodes, so that the integrity check still has zero additional delay under high-concurrency write load, achieving synchronous loss tolerance recovery and real-time check.

[0041] The bandwidth-aware replication pipeline dynamically adjusts the sending rate according to the link utilization rate and buffer depth, and the assembly line confirmation window ensures that the slices and Merkle roots are strictly sorted according to the write sequence number; the lake warehouse side solidifies the incremental snapshot through the snapshot segmentation function and the backtracking marker, and the segment-level hash allows differential scanning to judge integrity without full table traversal, and the cloud edge has a symmetrical view within a sub-second time window, upload-snapshot-checking enters a self-driven closed loop, completely eliminating the artificial reconciliation bottleneck.

[0042] The cloud differential scanner uses segment hash coarse screening combined with Bloom skip list fine screening to accurately locate the write sequence gap in constant memory access; the fast reconstruction engine uses slice positioning vector to once align the minimum data slice set, uses tensor core inverse matrix kernel to reconstruct the missing block in milliseconds, and uses grafting algorithm to only refresh the affected path of the Merkle tree, the whole gap discovery to closure process is less than single batch forward time, and the consistent dynamic self-healing is realized.

[0043] The online Merkle proof in the training phase uses tensor core idle time slot to calculate leaf hash, ring path buffer eliminates DMA fragmentation, and main thread completes sample level integrity verification without sensing; damaged samples are immediately pushed to the repair queue and replaced with a tensor generated by nearest neighbor prediction consistency interpolation, and the gradient stability gate ensures that the disturbance does not exceed the threshold and writes the repair entry into the repair log chain, both training throughput and accuracy are achieved.

[0044] After the training is completed, the frozen scheduler generates the evolution list in the write silent window and combines the snapshot root, repair root and evolution root into the frozen root through the sealing function, and then the redundancy recovery strategy only retains the minimum recoverable slice and deletes the check slice according to the exponential decay probability, and the write amplification feedback smooths the erase rate to prolong the service life of the solid state medium. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The data storage method flowchart for the artificial intelligence learning mode of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Please refer to Figure 1 The present application provides a data storage method for artificial intelligence learning mode, comprising,

[0048] In the artificial intelligence training scene driven by multi-source perception flow, the edge high-speed absorption layer Three challenges must be solved: first, the original samples from heterogeneous terminal devices are non-synchronous in time, and if there is no unified order, subsequent erasure coding cannot guarantee the block-level data correspondence; second, edge nodes often face link jitter and sudden failure , without a robust sequence replay mechanism, any single point failure will be amplified into a level replay error, directly destroying the time sequence consistency of the training data; third, cloud-edge cooperation requires continuous output of reliable sample blocks with timestamps in a high-concurrency environment , if the sequence control granularity is not fine enough, it will lead to the difficulty of incremental accounting of lake snapshot on demand.

[0049] Step one, assign a monotonically increasing write sequence number to the heterogeneous real-time stream on the edge side, and merge and splice it into a fixed-granularity sample block in the high-speed ring buffer according to the sequence number, while generating window start and end pointers, providing a nanosecond-level unambiguous time sequence coordinate reference for subsequent erasure coding, differential accounting and fault replay, and ensuring strict traceability of the link;

[0050] Step 101, use the write sequence number chain The arrival events of multi-source perception stream are monotonically numbered and merged in blocks Continuous sequence number samples are aggregated in real time to establish a block-level time sequence coordinate system that can be replayed, erasure-corrected and accounted;

[0051] Due to the exponential growth of the number of edge terminals , the mixing of raw stream inbound sequences can easily cause the row and column dislocation of the subsequent erasure matrix ; Therefore, by sequence number allocation-block alignment-persistent mapping three-stage linkage to smooth arrival jitter, it is ensured that any time sample block has a monotonically traceable time sequence reference; Thus, it ensures that the erasure coding only bears the loss tolerance function in the column dimension and no longer corrects the order; Further, when a node failure occurs, the replay pointer can directly locate the missing section according to the sequence number gap, and finally constrain the fault impact domain within the minimum block window .

[0052] In the design of the logical clock and the two-segment write sequence number generation function of the carry window size in the edge high-speed absorption layer :

[0053]

[0054] Among them: the logical clock : count frequency, value range , used to map physical time to discrete ticks; carry window size : window length aligned with block size, limit sequence number to monotonically increase within a single window, prevent 64-bit overflow;

[0055] Write sequence number chain : a monotonically increasing identifier at any time, ensuring that there will be no rollback due to multi-threaded concurrent writing.

[0056] Through the dual constraints of logical clock-window, sequence number generation only presents the increasing feature to the outside, and still has fine-grained time resolution to the inside, which can be directly used for subsequent Merkle digest positioning.

[0057] Designing a circular buffer and define the merge operator:

[0058]

[0059] where: : concatenation function, ensuring the sequence number metadata is atomically bound with the original payload;

[0060] , the fixed-granularity data unit formed by concatenating the sensed segments before the write sequence number;

[0061] , the globally monotonically increasing sequence number generated synchronously with the segment

[0062] , the th frame of original sensed data, which can be sourced from radar, camera, or inertial navigation, and has a variable byte length;

[0063] Block-level XOR merge , indicating dequeuing in ascending order; for each concatenation result, perform byte-level bitwise XOR in sequence, outputting a synthetic byte stream of the same length as a single concatenation result Block merge cache

[0064] : capacity of sample blocks, responsible for temporarily storing consecutive sequence number samples of the same window; Merge operator

[0065] : the merge operation result performed on the current sample block , taking a fixed-length byte sequence as the value, used as the subsequent leaf hash input; Merge upper limit

[0066] : the number of segments in the current block, determining the summation subscript end value; Through windowed merging, the number of rows of the erasure correction matrix

[0067] can be significantly reduced, improving the parallel SIMD encoding efficiency, while providing fixed-granularity check units for the comparison engine. To support accurate rollback within the fault domain

[0068] , introduce the replay pointer:

[0069]

[0070] where, is the ​​A sample segment is assigned a globally monotonically increasing sequence number in step one; For the confirmed persistent sequence number set, the success ACK sequence number set returned by the lake warehouse reconciliation, dynamic growth; replay pointer : points to the first missing sequence number; missing judgment field : all unsuccessful reconciliation sequence numbers.

[0071] The pointer locks the gap with O(1) search complexity, and the corresponding sample block can be quickly replayed combined with the merging cache, realizing millisecond-level data self-healing. By introducing a write sequence number chain in the edge high-speed absorption layer, block merging cache and replay pointer closed loop control, linear consistent timing order is realized without global lock in the multi-source heterogeneous perception scene, eliminating the block-level misplacement caused by terminal arrival jitter, so that the subsequent erasure matrix only needs to focus on the damage tolerance rather than the sequence, which significantly reduces the redundancy coding complexity; At the same time, the monotonic sequence makes any node failure only show as a sequence gap, which is quickly captured by the replay pointer , so as to converge the recovery range to the minimum block window.

[0072] Step 102, with timestamp block index As a link, the block in the block merging cache is mapped to the asynchronous erasure coding pipeline, ensuring that the block content and timestamp-sequence metadata are inseparable, realizing the natural sequence consistency of erasure coding and integrity checking.

[0073] When a single large block enters the erasure encoder, if there is no timestamp index, the reconciliation engine cannot quickly locate the corresponding fast line in the lake warehouse time travel interface. Therefore, this step inserts the timestamp-sequence double index , in the block-level erasure coding-Merkle root storage-temporary cache disk three-stage stream; Therefore, it eliminates the reconfiguration overhead between cross-node redundant slices and local blocks; So it can still maintain a highly sequential and consistent push behavior when the bandwidth is limited; Then the cloud training scheduler can directly scan the difference according to the index chain; Finally, the edge view and the main lake view are seamlessly spatiotemporally aligned.

[0074] In order to perform erasure correction without blocking the absorption layer, a priority queue based on the scheduler is used, and its scheduling priority function is:

[0075]

[0076] Where: merging priority : represents the priority score of the sample block in the asynchronous erasure coding queue. The larger the value, the higher the priority of being encoded;

[0077] Arrival interval : Time difference between two adjacent merged blocks arriving at the queue; Block loss rate : Sampled block loss rate of the source node within the history window, range : Weight coefficient , : Satisfy , dynamic adaptive, initial , ;

[0078] The function takes into account real-time (arrival interval) and reliability (block loss rate), so that the queue prioritizes protecting volatile blocks when the link is limited, and the weighted logic avoids encoding starvation caused by single indicator deviation; for each erasure-coded slice Construct hash leaf , then calculate the Merkle root from bottom to top:

[0079]

[0080] Where: root calculation operator , refers to the recursive process of pairing input hash list two by two, layer by layer hashing until a single root value is obtained according to the rules of the Merkle tree; Total number of slices;

[0081] Slice : Erasure coding output fragment; Timestamp index; Write sequence number chain; Hash function : BLAKE3, output 256bit; Merkle root : Full integrity fingerprint of the slice, used for subsequent playback verification.

[0082] Write the write sequence number chain and the timestamp index into the hash leaf, ensuring that any replay or splicing operation must match the sequence number-time stamp double index, eliminating the rearrangement attack surface under hash collision.

[0083] Write in the form of key-value pairs to the solid-state cache , the key is:

[0084]

[0085] Where: solid-state cache : NVMe array, key : Base64 encoded sequence number-time stamp string, ensuring uniqueness; value : Contains block payload and Merkle root.

[0086] Key uniqueness enables differential scanning to locate blocks with hash index only, and fast comparison of edge snapshot and cloud snapshot difference with time travel interface.

[0087] Global monotonic write sequence number chain built in step 101 - Block merge cache - Replay pointer Ternary closed loop, this scheme has laid the traceable sequential coordinates for multi-source perception stream; and step 102 further indexes the timestamp with Merkle root Deep binding to erasure coding slices, atomic level coupling of data integrity and sequential consistency is achieved. The two steps are logically in front of and behind each other: the continuous blocks and their sequence number chain output by the former are directly used as input for the latter to generate redundant slices and Merkle root; the latter in turn checks the sequence number chain continuity in reverse with the Merkle root to form a closed loop correction.

[0088] Atomic binding of timestamp index and Merkle root in erasure coding link makes integrity verification and sequential consistency naturally homologous, fundamentally avoiding the classic paradox of blocks being restored but unable to align snapshots. The asynchronous erasure coding queue driven by priority queue automatically protects volatile blocks under bandwidth constraints by dynamically balancing real-time and reliability, breaking the rigid shackles of traditional static redundancy ratio. Further, the local solid-state cache uses unique mapping based on sequence number-timestamp key, enabling differential scanning to no longer rely on complex directory traversal to complete lake warehouse reconciliation, providing a lightweight, traceable snapshot path for super large edge nodes.

[0089] In step one, write sequence number chain , block merge cache and timestamp index have laid unambiguous and traceable sequential coordinates for each sample block ; however, only when these blocks are further transformed into cross-node redundant slices and accompanied by tamper-proof integrity fingerprints, can the entire cloud-edge training pipeline truly resist node disconnection, disk damage, and link jitter and other multiple failures.

[0090] Step two unfolds in this context: on the one hand, it performs asynchronous erasure coding on sample blocks without blocking the absorption layer write, generating a mixed set of data slices and check slices , so that the original block can be quickly restored from any slice in the event of node failure; on the other hand, it needs to calculate the Merkle root synchronously, and write the slice-root pair to the solid-state cache with key To provide a unique source for subsequent differential scanning and auditing. This process not only makes full use of the parallel encoding capabilities of edge nodes, but also ensures that metadata and payload are atomically persistent, avoiding the chain coupling delay caused by traditional encoding first and then checking.

[0091] Step two, while maintaining zero blocking of write threads, use priority queue scheduling to perform adjustable redundancy erasure coding on the ordered blocks and synchronize the calculation of the Merkel root, atomically write the slice and root as key-value pairs to the cache, taking into account node fault tolerance recovery and block-level integrity checking, while preparing for subsequent bandwidth-aware replication and differential scanning by preparing cross-node redundancy, hash index, and timestamp double-index mapping;

[0092] Step 201, for each sequential sample block , build an asynchronous erasure coding pipeline that outputs a set of cross-node redundancy slices while maintaining zero blocking of the absorption layer , and dynamically adjust the number of data slices and the number of check slices ;

[0093] The absorption layer output rate is much higher than the single-node storage bandwidth. If synchronous erasure coding is directly performed, it will inevitably lengthen the write path and break the monotonicity of the write sequence number chain . Therefore, the encoding needs to be placed in an asynchronous thread pool and rate-shaped with a priority queue ; thus ensuring that write threads are only responsible for pushing sample blocks to the encoding buffer , while the actual matrix operations are completed in the background; then, by adjusting the redundancy , the proportion of check slices can be reduced when bandwidth is tight; then, using the Rabin-Karp rolling hash algorithm, the slices are evenly distributed to different nodes to avoid single-machine hot zones; finally, even if multiple nodes fail, as long as any slice is retained, the original block can be recovered, achieving controllable fault tolerance.

[0094] To quickly switch the redundancy level under different loads and fault probabilities, a parameterized Vandermonde matrix is used, where the redundancy ratio is adjustable in real time:

[0095]

[0096] Where: the number of data slices : the value range , the minimum number of recoverable slices required for encoding; the number of check slices : the value range , used to resist node failures; the redundancy ratio : the value range , representing the fault tolerance level of redundancy; : Galois field multiplication primitive, taking values in : primitive element in GF(2^m); erasure code matrix : generation matrix for determining slice linear combination coefficients.

[0097] By adjusting redundancy ratio in real time , the same encoding core can switch redundancy in milliseconds, avoiding the traditional scheme from pausing writing due to matrix reconstruction. After the slice is generated, it must be distributed to multiple edge nodes. If only round-robin writing is used, it is easy to cause the bandwidth bottleneck node to become a single point of recovery. The Rabin-Karp rolling hash algorithm is used to select the next target node according to the real-time link margin Exponential decay selection:

[0098]

[0099] Wherein: represents the slice transmission probability from the source node to the target node ;

[0100] Slice : represents the i-th source node, any segment from the sample block ; candidate node : represents the i-th target node, a storage node in the edge cluster; link margin : and the encoding node point-to-point available bandwidth; represent the bandwidth freedom of the target node , and the cycle index ; temperature coefficient : , adjust the distribution tilt, default 0.3;

[0101] This soft distribution strategy tilts more slices on high-bandwidth nodes while maintaining probabilistic balance, avoiding the formation of fixed hotspots. To reduce the foreground writing pressure, the verification slice can be generated by the background refresh thread with a delay, as long as it is written before the lake warehouse time travel reconciliation trigger:

[0102]

[0103] Wherein: snapshot deadline : the latest time point that the current incremental snapshot needs to be completed; encoding completion time : the time when the data slice generation is completed; maximum delay threshold : used to limit the background delay; delay window : the maximum tolerable delay for the background thread to generate the verification slice.

[0104] ​​The delay generation of the check piece can release the encoding computing power during the peak period, and smooth the redundancy overhead to the link idle period.

[0105] In the asynchronous pipeline mode, the adaptive erasure correction matrix, bandwidth-aware distribution, and background check piece refreshing triple mechanism are introduced, so that the write thread of the edge node always remains in a non-blocking state. Even if there is link jitter or node disconnection, the minimum piece set required for recovery can be dynamically matched and the probability-based piece dropping strategy can be used to limit the recovery to within the preset threshold, showing the innovative ability to maintain linear expansion in a fluctuating resource environment. At the same time, since the background thread can delay the generation of the check piece, it can prioritize business timeliness during the peak write period and supplement the redundancy during the idle period, thereby balancing energy consumption and hardware wear and tear throughout the life cycle.

[0106] Step 202, for each slice set Instantly generate Merkle root , and take the key Atomic write slice-root pair to solid-state cache , so as to integrate integrity verification and snapshot index.

[0107] Only redundant slices are not enough to ensure end-to-end trust in the training phase. When any piece is tampered with or out of order in the link, the recovery may be successful, but the semantics have changed. Therefore, each piece needs to participate in the hash together with the order metadata of the block it belongs to; so that any single-byte disturbance is mapped to the Merkle root change; and the root and key are persisted together to ensure that the root is obtained by searching the key; then only the root needs to be compared on the lake warehouse side to determine the block-level consistency; finally, large-scale differential scanning is completed with the minimum bandwidth.

[0108] To ensure the embedding of sample timing properties integrity fingerprints, the write sequence number chain is spliced with the timestamp index to each slice hash original image to ensure that any rearrangement will cause the root to change and eliminate out-of-band replacement attacks:

[0109]

[0110] Where: leaf node : Hash value of slice ; Hash function : BLAKE3, output 256bi; concatenation operator : Byte order splicing operation.

[0111] When the number of slices is large, a single-layer Merkle tree will cause the root update cost to grow with ; the scheme uses a two-level grouping tree to reduce the recalculation overhead:

[0112]

[0113] wherein: group number : slices are divided into subgroups; group size : number of slices in each group; group root : subgroup root of the first group; global root : final Merkle root of the block , used for cloud-edge consistency check.

[0114] wherein, the hierarchical tree makes it only need to recalculate the group root when rewriting local slices, and then update two layers from top to bottom, significantly reducing the algorithmic consumption of low-frequency recalculation. Write the root global root and slice index to the solid state cache , use the symmetric two-phase commit to prevent partial failure:

[0115]

[0116] wherein: pre-write : write to a temporary page and mark the unsealed state; seal : flush the index after the NVMeFTL has been marked as persistent; rollback : delete incomplete entries; key : Base64 string, uniquely identifies the block ;

[0117] The two-phase commit ensures that even if the power failure occurs in the intermediate state, there will be no inconsistency of storing the root but missing the slice or storing the root and missing the slice, which anchors the minimum transaction unit of the lake warehouse time travel interface that can be replayed.

[0118] By embedding the write sequence number chain and timestamp index directly into each slice hash image, step 202 completes the integrity binding at the moment of slice generation, making the order the fingerprint a natural attribute; then supplemented by a hierarchical Merkle tree to reduce the recalculation range of local rewriting, and with the key as the only anchor to complete the two-phase commit, finally realize the key-root-slice trinity of the minimum transaction snapshot. This not only makes differential scanning only need to compare the root to assert the block-level consistency, but also flattens the traditional separate data-first and then check serial process into the same completion, greatly shortening the reconciliation window.

[0119] Step two, through step 201, enables the edge node to not block the write sequence number chain On the premise, real-time output and network status synergistic change of redundant slice set; subsequent step 202 further with serial number-time stamp hybrid hash-layer Merkle tree-key value type atomic commitment ternary closed loop, integrity fingerprint and snapshot index binding as a single transaction.

[0120] The former output slice Directly into the hash leaf generation of the latter, the latter calculated root In turn for the former asynchronous slice refresh provides re-computable verification path. Compared with the traditional first full block redundancy and then offline verification mode, the slice generation is accompanied by a verifiable fingerprint, forming a synergistic effect of redundancy and verification.

[0121] After step two, the slice-root pair is atomically persisted in the local solid-state cache , these verified and bound data units must be safely and efficiently migrated to the cloud-side object storage in order to generate incremental snapshots that can be referenced by the lake warehouse time travel interface and provide a basis for subsequent differential scanning. Due to the fact that the access network where the edge node is located is often in a mixed state of bandwidth mutation and latency jitter, if the traditional static batch upload mode is still used for pushing, not only is it easy to form write amplification, but also the sequential drift will destroy the monotonic and traceable characteristics of the write sequence number chain .

[0122] Step three, push the slices and Merkle roots to the object storage in write sequence order through an adaptive rate replication pipeline, use the flow confirmation window and global version counter to ensure multi-threaded order preservation; then call the lake warehouse time travel interface to solidify the incremental snapshot in a diagonal balanced segment and write the traceable marker, so that the edge view and the main lake view are consistent within sub-second and provide segment-level fingerprints.

[0123] Step 301, in the bandwidth dynamic fluctuation scenario, build an adaptive replication pipeline so that the slices and Merkle roots can be continuously pushed to the object storage in write sequence order , while reducing the RTT constraint on throughput through the flow confirmation window.

[0124] The instantaneous available bandwidth of the edge-cloud link is strongly related to the queuing delay , when the replication rate is too fast, queue congestion will drag the tail delay and cause out-of-order retransmission; on the contrary, it also wastes peak bandwidth. Therefore, the replication rate needs to be adjusted based on the real-time link utilization and local replication buffer depth ; so as to automatically enter the conservative window in high load and the detection window in low load; and then cooperate with the sequential confirmation window Guarantee the arrival order of replication ACKs does not break the write sequence chain; then use a monotonic version index on the object storage side Take the sequence of the sequence number, avoid the reverse coverage of the write first and the version number later; finally, even if the link occurs a sharp drop from 5Gbps to 200Mbps, the replication rate and the order control can be self-stabilized without manual intervention.

[0125] Replication rate Link utilization And buffer depth Take the input, keep high throughput and low out-of-order through the double ring adaptive function:

[0126]

[0127] Where: available bandwidth : the instantaneous measured link free bandwidth, range , used as the upper limit of the rate;

[0128] Link utilization : Real-time ratio, dimensionless, 1 indicates congestion; adaptive coefficient : , adjust the Sigmoid convergence slope; buffer depth : the current number of bytes to be sent in the replication buffer, range ; maximum buffer : fixed upper limit, typically 512MB; suppression coefficient : , determine the reduction amplitude of the buffer water level to the rate; replication rate : the current pipeline data transmission rate, unit Gbps.

[0129] Two-factor coupling suppression makes the rate quickly smooth down once it approaches saturation, and quickly releases bandwidth when the buffer is low and the free is high, maintaining queue isochronous stability; to weaken the RTT jitter, use window size Flow water confirmation, window sliding is pushed by the minimum unconfirmed sequence number .

[0130]

[0131] Where: maximum window : Slice, limited to the number of slices; maximum sent sequence number : the maximum write sequence number that has been pushed but not necessarily confirmed; minimum unconfirmed sequence number : the left boundary of the window, which moves to the right after the ACK arrives; window size : the number of allowed unconfirmed fragments.

[0132] The window ensures that the ACK does not disrupt the write sequence number chain even when it arrives out of order, and only slides as a whole when the leftmost gap is acknowledged, embodying the sequence number priority principle; the object storage version table is indexed in ascending order Mapping to key The index generation function relies on a global atomic counter :

[0133]

[0134] Wherein: atomic counter : version number of the last successful write on the object storage; current version index : incremental number generated this time, 64 bits, ensuring monotonicity; key : , uniquely identifies the slice-root pair.

[0135] The global monotonic version number carries the write sequence number, making the object storage naturally ordered; even if multiple threads are concurrently PUT, there will be no overlap or reverse order visibility.

[0136] The adaptive replication pipeline integrates transient control of replication rate, order protection of pipelined acknowledgment window, and persistent mapping of monotonic version index, breaking through the traditional constant-speed upload + batch ACK limitation that easily produces retransmission storms in narrow-band sudden drop scenarios. By allowing rate adjustment to refer to the dual feedback of available bandwidth and buffer depth , it autonomously forms a light-probing and fast-converging link self-healing rhythm without relying on external congestion protocols; and the acknowledgment window only slides as a whole when the smallest unacknowledged sequence number is acknowledged, ensuring that any ACK out of order will not disrupt the consistent playback of the write sequence number chain. Further, the monotonic index treats the object storage as a topologically transparent sequential baton, eliminating race write conflicts between PUTs on different nodes, achieving multithreaded safety without sacrificing concurrency.

[0137] Step 302, call the lake warehouse time travel interface, perform incremental snapshot persistence on the latest upload version number interval , and implant a traceable marker , so that the edge view is consistent with the main lake view;

[0138] Although the object storage is already ordered, if the snapshot is not solidified in time on the lake warehouse side, the cloud scheduler cannot locate the gap through differential scanning in step four. Therefore, the time travel interface needs to be triggered immediately after each version number window is capped; thus, the current version index is mapped back to the original write sequence number and timestamp index interval;

[0139] Further, the snapshot segmentation function Generate segment metadata by span of write order, avoid metadata explosion caused by huge list; then insert a traceable marker at the end of each segment , containing the Merkel root , version range and CRC32 check, ensure that a single segment can be verified independently; finally, when the cloud scheduler listens to a new traceable marker event , it can directly load the corresponding segment for differential, without the need for full lake scanning.

[0140] According to the span of write order And version span Adaptive segmentation, the formula of equal angle makes the influence of write order and version number equal, so that the size of the segment changes smoothly with the growth rate of the two, preventing exponential explosion at one end:

[0141]

[0142] Among them: write order span : the maximum sequence number minus the minimum sequence number in the capped window; version span : the difference between version numbers in the same window; scale factor : , used to control the granularity of the segment; the number of segments : the number of snapshot segments obtained.

[0143] A traceable marker is constructed at the end of each segment , the content of which is compressed by hash , generating the formula:

[0144]

[0145] Among them: Merkel root : the root fingerprint of the last block in the corresponding segment; starting version : the starting version number of the segment; end version : the end version number of the segment; block : the cyclic redundancy check code of the block payload; hash function : BLAKE3, consistent with the previous; traceable marker : segment-level metadata anchor, 256 bits;

[0146] The marker encapsulates the three-dimensional properties of order, version and content at a time, and the cloud only needs to sign the traceable marker To determine the consistency of the segment level, greatly compressing the I / O of the account. After the snapshot is persistent, the lake warehouse returns a confirmation vector to the edge, so that the set of solidified version numbers is added to the confirmation set :

[0147]

[0148] wherein: validation vector , set of version numbers that have been validated by the lake; validation set : defined in step one, used for replay pointer , , indicates that the data or data block has been validated;

[0149] Bidirectional write-back reports the main lake validation to the edge in real time, making the replay pointer Closed loop is also obtained at the version level, avoiding repeated uploading or mistakenly deleting un-solidified segments.

[0150] Lake snapshot solidification process takes snapshot segment number As the core of granularity control, by diagonally coupling the write sequence number span and the version span , the system automatically balances between metadata and snapshot size, so that the incremental persistence is neither excessively fragmented nor inflated; then, with the help of the embedded traceable marker , the Merkle root , version interval and CRC are encapsulated once, and the lake only needs to verify the single segment fingerprint to confirm the data integrity, fundamentally avoiding the expensive I / O of long chain directory comparison. Subsequently, the lake writes back the validation vector to the edge, directly driving the replay pointer of step one, closing the corresponding gap and forming a timely closed loop to prevent redundant repeated uploading.

[0151] The replication rate - pipelined validation window - monotonic version index Three-way pipeline, the system can still push slices in write sequence number order without obstruction under strong fluctuating links; and step 302 further uses the snapshot segmentation function - traceable marker - bidirectional consistent write-back three-segment closed loop, which maps the sequential version in object storage to the incremental snapshot that can be referenced by the lake time travel, and absorbs the confirmation result back to the layer.

[0152] The two steps are closely related: the continuous version number interval generated by the former is directly input into the snapshot segmentation algorithm of the latter, and the validation vector rewritten by the latter converges the replication window in real time, forming a push-solidification-validation self-driven closed loop. Compared with the traditional batch uploading + offline snapshot mode, this scheme outputs verifiable version numbers at the moment of uploading and writes back validation vectors at the moment of snapshot solidification, so that the end-to-end consistency delay of cloud-edge is compressed from minutes to sub-seconds;

[0153] Meanwhile, the traceable marker and the Merkle root Binding, let the differential scanning only need to process segment-level fingerprints to locate the gap, provide a millisecond-level starting point for the fast reconstruction of step four.

[0154] In step three, the edge node has copied the rate , version index And the retroactive marker Push the slice-root pair to the object storage and solidify it as an incremental snapshot; however, this process can only guarantee the success of writing, not the immediate readability of the content by the model. Before truly entering the cloud training pipeline, there are still two potential faults: first, although the snapshot is formed, its sequence number chain Can appear gaps due to link jitter, if not discovered in time, it will trigger I / O blocking during training; second, the redundant slice has been written, but if the cloud does not rebuild the missing block in the first time and refresh the time travel index , the next sampler will have difficulty playing back sequentially from the continuous address space, causing the gradient descent process to stall.

[0155] Step four, the cloud training scheduler listens to the segment-level marker event, first uses segment hash and Bloom-skip list structure to quickly differential scan the write sequence number chain, locates the gap, and then uses the slice positioning vector to align the minimum data slice set once, uses GPU inverse matrix to quickly rebuild the missing block and graft the refreshed time travel index and global Merkle tree, realizes the closure of the gap as soon as it is discovered, and guarantees the consistency of the training data.

[0156] Step 401, trigger the differential scanner in an event-driven manner, locate all the gap sets in the write sequence number chain in the shortest time, and output the missing segment tuple that can be directly called for the subsequent reconstruction module.

[0157] The generation frequency of incremental snapshots is higher than the cloud batch training beat, if the write sequence number chain is still compared by full table scanning, it will cause the metadata scanning to drift in the TB level range due to the mixed arrangement of hot and cold areas; therefore, the previously written retroactive marker must be used as a logical anchor point to first coarsely lock the possible gap segment, and then do fine-grained skip list scanning within the segment; thus, the original O(N) full comparison is reduced to O(logN+|gap|) sparse comparison;

[0158] Then use one-time hash to determine whether the segment is complete, and only when it is not complete, enter the sequence number level jump; then write the discovered gap into the gap set and broadcast it to the reconstruction scheduler; finally form a three-segment pipeline of segment fingerprint→sequence number skip list→gap set, so that the differential positioning matches the upload rate, without dragging the GPU standby time.

[0159] To quickly rule out most complete segments, the differential scanner first calculates the segment-level hash for each new segment :

[0160]

[0161] Wherein: backtracking mark : Segment fingerprint inserted in the previous step, 256 bits. Segment end Merkle root : Integrity fingerprint of the last block in the segment, 256 bits; Hash function : BLAKE3, constant; Segment-level hash : Fast fingerprint for comparison with historical snapshots.

[0162] If the segment-level hash is consistent with the lake warehouse inventory record, it can be concluded that the segment is complete, and the serial number-level scanning is directly skipped; otherwise, the next technical point is entered, and the differential scanner establishes a two-level structure for the write serial number chain in the segment that needs fine-grained comparison:

[0163]

[0164] Wherein: : indicates the sequence Bloom filter is applied to store the existence of serial numbers with a 0.01 false positive rate;

[0165] Skip array : indicates the use of Skip List structure, one index is built for every 8 serial numbers, O(1) jump;

[0166] Binary structure : indicates a composite value composed of two data structures, which guarantees low memory and provides random access;

[0167] In the serial number dense segment, through Bloom filtering of the large segment existence area, and then using Skip to accurately locate the missing number, the gap discovery is completed with 2 memory accesses. After the missing number is found, the following function is used to construct the gap priority :

[0168]

[0169] Wherein: serial number span : Gap length, unit piece; Time aging coefficient : Gap distance from the current time in seconds; Weight , : satisfies , default 0.6, 0.4; Gap priority : The larger the value, the sooner it is reconstructed.

[0170] Taking into account both gap length and time aging factor, this approach avoids small but outdated gaps remaining unused for extended periods, while also preventing new, large gaps from slowing down training. Through a delicate three-stage chain of tag hashing, skip lists, and priority, the petabyte-level snapshot differencing is reduced to memory-level computation, providing a precise and well-sorted gap set for step 402. .

[0171] The differential scanner, through a three-stage pipeline of prior screening, skip list indexing, and gap priority, reduces the snapshot verification of a petabyte-scale lake warehouse from linear traversal to fingerprint localization plus sparse jumps. First, segment-level hashing... This allows the system to exclude the vast majority of complete segments with minimal metadata reads, significantly reducing disk addressing and cache thrashing; subsequently, binary skip lists... By limiting the resident memory overhead to a fraction of the total number of sequence numbers, it can accurately locate missing numbers within two memory accesses, overcoming the traditional limitation of Bloom filtering that can detect but not locate; ultimately, gap priority... By introducing a time aging parameter, the reconstruction scheduling and training cycle are tightly coupled, thus avoiding order starvation phenomena such as shortage starvation and long-term outstanding small gaps. After the three work together, the differential-location-dispatch link enters a sub-second closed loop, allowing cloud resources to complete differential processing within GPU idle slots without blocking iterations.

[0172] Step 402: Use the gap set output by the differential converter Call the fast rebuild engine From redundant slice set with minimal I / O Reassemble missing blocks And refresh the time travel index. With Merkel tree This ensures continuous consistency between the cloud and the edge.

[0173] The reconstruction process needs to be fast while ensuring that the snapshot version consistency is not compromised; relying solely on conventional RS decoding will result in multiple remote reads due to the wide distribution of fragments; therefore, fragment location vectors are introduced. This allows for the aggregation of sufficient fragments in a single step, achieving minimal replication and one-time alignment, and subsequently employing a GPU matrix core. Parallel inverse operation, solving the original block in tens of milliseconds; then using block-level root... Then, a global Merkle tree is embedded, and only affected paths are updated using the forest-grafting algorithm; finally, the updated time travel index is written back. and segment markers This signifies that the gap has been closed.

[0174] Generate slice location vector :

[0175]

[0176] wherein: slice positioning vector : list of nodes that need to be touched by one remote call;

[0177] node : edge or cloud storage node that stores a slice; slice : data or check slice corresponding to a missing chunk; minimum number of data slices : defined by step two, 4-16 slices;

[0178] Execute on tensor core:

[0179]

[0180] wherein: inverse matrix : corresponding redundancy ratio : Vandermonde inverse, dimension ;

[0181] Truncated slice vector : according to slice positioning vector : aggregated minimum data slice.

[0182] Reconstructed chunk : original byte sequence of the missing chunk, representing the final chunk data. It is the result of applying the inverse encoding function on some input.

[0183] GPU performs matrix-vector multiplication, which can be completed in about , and the latency is much lower than that of CPU slicing and reorganization. The execution time travel index and Merkel tree incremental refresh, the refresh operation follows:

[0184] ,

[0185] wherein: key : associated reconstructed chunk; index : lake warehouse time travel index, mapping write sequence number to object URI; grafting function : only recalculate node hash in the affected path of the Merkel tree; global tree : top-level Merkel tree covering all chunks in the current training period;

[0186] Incremental mode avoids recalculating the entire tree, keeping lake warehouse query response time from being slowed down by reconstruction operations, while the index can be read by training at once. The slice positioning, GPU decoding and metadata refresh are woven into the shortest path of positioning-decoding-grafting, so that the gap closure delay converges to low milliseconds, thus providing seamless continuous data plane for sample playback in step five.

[0187] The fast reconstruction engine utilizes slice positioning vector Minimizing data fetches with a single GPU inverse matrix High concurrency decoding with grafting functions Only recompute the affected path's Merkle tree nodes, and the entire gap filling process almost does not touch irrelevant cold data. This design can complete cross-node aggregation in a single I / O, avoiding the multiple rounds of pulling required by regular RS decoding; and GPU operations reduce the slice decoding delay to the order of hundreds of microseconds, far lower than the millisecond scale of CPU-level delays. More importantly, incremental grafting allows the global tree to maintain a stable root, so that the entire tree does not need to be rehashed due to local reconstruction, thereby ensuring time travel indexing Read immediately after filling.

[0188] Step four with the marker hash of step 401 Bloom-Skip skip list Priority function Three-stage differential link, high-speed positioning of write sequence gaps in the rapidly changing cloud-side lake warehouse; then step 402 uses slice positioning vectors GPU inverse matrix Grafting refresh Three-stage reconstruction pipeline, gap segment filling and real-time updating of index and Merkle tree, two steps tightly connected and complementary.

[0189] Compared with traditional offline verification + batch reconstruction, the combination of segment markers and sequence index depth allows for sparse positioning at the logical level and batch reorganization at the data level, greatly shortening the differential-reconstruction closed-loop path; at the same time, through the combination of GPU decoding and incremental grafting, high-throughput matrix calculation and low-overhead metadata updating are parallelly promoted, avoiding the broken state where reconstruction is complete but immediate querying is not possible. This makes cloud-edge views enter a truly continuous consistent state: any new snapshot will be verified and completed within sub-seconds, and training schedulers can safely pull samples without explicit waiting.

[0190] In step four, the cloud training scheduler has ensured the lake warehouse time travel index with the global Merkle tree without gaps; however, this only completes data face continuity.

[0191] Once the data is read by the training process, it may still introduce fine-grained damage in the sampling-loading-forward link due to network flips, disk flips, or GPU memory soft errors. Traditional deep learning frameworks mostly use the strategy of skipping or filling 0 for failed reading, which may cause gradient shock at the light end and introduce hidden bias at the heavy end, especially in self-supervised or contrast learning scenarios, where the dependence on sample integrity is higher than in regular classification tasks.

[0192] ​Step five, use GPU-CPU cooperation to generate sample Merkle proof in training sampling to check integrity in real time. Any mismatched sample is written into the damaged queue and an alternative tensor is generated by passing through the neighbor prediction consistency interpolation in the same batch. Gradient stability gate monitors disturbance in real time and adjusts weight. Repair events are written into the repair log chain and the repair log root is updated to realize online self-healing and auditable trace.

[0193] Step 501, in the training sampler Read each sample Synchronously generate a Merkle proof And perform integrity checking; if the check fails, immediately determine it as a damaged sample and trigger the uplink repair pipeline.

[0194] In the GPU training link, I / O bandwidth and computing power often have a scissors difference; synchronous verification will block reading if the throughput is reduced to a single thread level; therefore, the hash path inference and GPU decoding flow are parallel, and the hash is calculated using the TensorCore idle period; thus, real-time verification is completed without reducing GPU utilization; and through segmented hash path buffering Splice multiple proofs in batches to reduce PCIe back and forth; then, when the check fails, add the sample to the repair queue With write serial number Finally, maintain the training main thread to continue without awareness, while the background repair thread takes over the interpolation logic.

[0195] Three-line description--to reduce PCIe round trip, hash path calculation is divided into GPU-level leaf hash And CPU-level parent hash Two levels:

[0196] ,

[0197] Among them: sample data : normalized Tensor, resident GPU; write serial number chain : generated in step one, 64 bits.

[0198] Timestamp index : 64 bits; leaf hash : GPU end calculation result, 256 bits; sibling hash : retrieved from segmented hash path buffer Parent hash : CPU end synthesis, push Merkle path up.

[0199] ​GPU completes high-throughput BLAKE3 leaf hashing and returns deep pipelining, CPU only concatenates the last layer, and NVLink P2P minimizes cross-bus data transfer. The complete proof is split into fixed 1KB segments and placed in a ring buffer for GPU-side batch pulling, solving the DMA fragmentation caused by random path length.

[0200] Buffer capacity configuration:

[0201]

[0202] where: batch size : current GPU small batch size; multiplication coefficient : , to ensure that the CPU can be completed before the buffer overflow; ring buffer : manage parent hash fragments.

[0203] The ring design rotating by batch can ensure lock-free concurrency of the buffer under large batch reading. The CPU only needs to fill by segment, and the GPU side DMA takes the required path once. Once the parent hash is pushed to the root still not match the global root, generate:

[0204]

[0205] and write to:

[0206]

[0207] where: error description : record the first layer mismatch hash; damage mark : single sample integrity failure event; repair queue : FIFO capacity 10K, used for asynchronous interpolation threads.

[0208] Use the queue to transfer the failed samples of the synchronous check to the background. The main training thread only sets the damaged sample mask to 0 and continues gradient accumulation. Step 501 realizes the lightweight pipeline of verification parallel and exception hosting through GPU-CPU collaborative hashing, ring path buffer and damage queue isolation, creating a time window for subsequent interpolation.

[0209] Make the overhead of sample-level integrity verification almost equivalent to normal tensor loading: leaf hashing is completed on-chip when TensorCore is idle, and does not occupy the main computing power; parent hash concatenation can converge in CPU L3 cache without touching remote DRAM; ring buffer cuts the indefinite length Merkle path into fixed length segments, completely eliminating DMA fragmentation. At the same time, any failed verification is marked as damaged and pushed into the repair queue immediately , the main thread just need to zero the mask and continue gradient accumulation, corrupted samples are dragged out of the main batch, the disturbance to training distribution is minimized, and a millisecond window is won for subsequent interpolation.

[0210] Step 502, perform prediction consistency estimation on each corrupted sample in the repair queue Step 503, perform prediction consistency estimation on each corrupted sample in the repair queue Step 504, generate a replacement tensor through online interpolation and synchronously update the repair log , ensuring continuous gradient flow and audit chain closure.

[0211] Directly discarding bad samples will cause fluctuations in the number of valid samples within the batch, causing BN / LayerNorm statistical drift; therefore, a replacement tensor needs to be generated within the same training iteration; interpolation is performed based on the nearest neighbor time window and the current EMA of the model; then, after interpolation is complete, the repair event is written to the log Merkle chain , and the repair log root is backfilled to mark the segment; then, it ensures that during auditing, the same Merkle path can be used to verify that the sample has been repaired and the repair strategy is transparent; finally, it stabilizes the gradient without introducing black-box changes.

[0212] The nearest neighbor window prediction consistency estimation algorithm is as follows:

[0213]

[0214] Where: the nearest neighbor set : the 4 samples closest in time to the replacement tensor ; the replacement tensor : is used for forward propagation and gradient backpropagation;

[0215] The weight : , the sum after normalization is 1; the normalization factor : ; the exponential moving average : the average of the model's previous forward propagation output; the interpolation balance coefficient : , against overfitting;

[0216] The nearest neighbor weight is inversely related to the sample-mean distance, ensuring that the replacement tensor does not deviate from the distribution; the interpolation balance coefficient injects global smoothing, reducing small dataset noise. After interpolation, the disturbance to the gradient distribution needs to be evaluated, and the gradient disturbance :

[0217]

[0218] If ​Then the interpolation balance coefficient is adjusted And recalculate;

[0219] Wherein: the current gradient : the backpropagation gradient containing the replacement tensor; the reference gradient : the pre-replacement gradient snapshot; the stability threshold : ; gradient disturbance : relative change rate

[0220] Wherein, the dynamic gate ensures that the interpolation strategy does not bring 15% gradient explosion, keeping the training smooth. Repair entries are generated for each interpolation :

[0221]

[0222] And add repair logs :

[0223]

[0224] Wherein: timestamp : repair completion nanosecond timestamp; repair entry : single event fingerprint; repair log : collect all repair events in order; repair log root : log Merkle tree root, 256 bits

[0225] Write the repair log root Back to the segment marker, so that the auditor can verify the repair record and replacement value with only the root, forming a data-repair-proof trinity chain.

[0226] Step 502 realizes the full closed loop of damage-repair and repair-verification through consistent interpolation prediction, gradient stability gate and repair log chain, and simultaneously closes the loop at the parameter layer and the audit layer.

[0227] Generate replacement tensors within the distribution with batch-in near neighbors and model sliding mean , ensuring that the number of effective samples within the same iteration remains unchanged, and BatchNorm mean and variance do not need to be re-estimated; gate quantity gradient disturbance Real-time detection of the impact of interpolation on gradient, if over threshold, immediately adaptively converge parameter interpolation balance coefficient , avoid repair over-repair and introduce gradient explosion; each interpolation is also Written to the Merkle-ized log, producing a repair log root And backfill the segment marker, so that the auditor can verify the repair details with only the root.

[0228] On one hand, GPUHash computation takes advantage of TensorCore spare capacity to compress integrity check cost to microsecond level; on the other hand, dynamic feedback of interpolation-gradient gating ensures that the disturbance of repaired samples to gradient flow is under threshold, maintaining the stability of optimizer. Furthermore, repair log chain embeds repair into the same Merkle domain, providing a self-consistent and non-repudiable event sequence for the evolution freeze in step six.

[0229] Under the serial promotion of the first five steps, the full-link closed loop from edge collection, erasure coding, snapshot into lake, differential reconstruction in the cloud to online repair during training has been completed; however, the end of training does not mean the natural end of the data life cycle. At this time, there are still three types of subsequent requirements that have not been solved: first, regulatory compliance-the snapshot versions accumulated during training, repair logs and operation events must be sealed as immutable evolution for subsequent audit or model accountability; second, resource controllable-as training terminates, the redundant slices over-provisioned in the encoding phase have lost the value of immediate recovery, and if not cleaned up in time, they will drag down the capacity budget of NVMe cache and object storage; third, traceable freeze-snapshot freezing must ensure that the data-repair-evolution three chains are atomically locked under the same fingerprint, otherwise subsequent cleanup may cause reference chain breakage.

[0230] Step six, generate the evolution list and synthesize the frozen root by sealing function with the snapshot root and repair log root in the write silent window at the end of training, then delete redundant check slices according to the retention probability function and use write amplification negative feedback to smooth erase, after completion, revise the version table to generate the cleanup root, and splice it with the frozen root to form the audit final root, making the archiving, cost recovery and traceable audit one-time closed loop;

[0231] Step 601, generate evolution record, atomically freeze the current snapshot state and lock the audit chain root, so that the snapshot content fingerprint , repair log root and evolution issue root converge into a unique frozen root , providing an irrepudiable reference for subsequent cleanup.

[0232] The end of training is often accompanied by high-frequency write-read mixed operations, and if directly frozen, it will compete with the tail batch gradient write for metadata lock; therefore, a freeze scheduler is introduced to enter the write silent window after listening to the training completion event; thus ensuring that all cache writes are landed; then based on the version index , the freeze vector is calculated and the write permission is locked; then the repair log root and the snapshot root are sequentially embedded in the evolution list, and the evolution issue root is calculated ; finally, the on-chain sealing function Synthetic frozen root , and broadcast to both cloud and edge.

[0233] Freeze vector generation and write quiesce window, where a freeze vector is introduced to evaluate whether freeze condition is met :

[0234]

[0235] where: active write delay : time difference between the last write operation in the past 30s and current time, in s; value .

[0236] Buffer residual depth : bytes of write buffer that have not been flushed to disk; flush completion rate : persisted data / data to be persisted, range ; freeze vector : evaluate whether write quiesce window meets threshold;

[0237] If , , are met simultaneously, freeze is triggered, where: quiesce threshold : ; buffer threshold : .

[0238] where, three-dimensional indicators jointly prevent tail batch gradient impact from grabbing lock freeze while still flushing, ensuring data consistency. After freeze condition is met, the scheduler generates evolution entries in write sequence order :

[0239]

[0240] and aggregates into evolution list :

[0241] ,

[0242] where: training timestamp : nanosecond time when the sample completes the last gradient update; evolution entry : single edition fingerprint;

[0243] Evolution list : complete version sequence at freeze time; evolution signing root : Merkle root of evolution list; is processed (such as hashing or encoding) for a slice to generate a root value ;

[0244] Thus, the version-time-sequence three-dimensional information is once fixed as an unforgeable list, providing hard evidence for investment accountability,

[0245] Final freeze root synthesis:

[0246]

[0247] Wherein: snapshot root : Global root of the last time travel snapshot; repair log root : Step five output;

[0248] Random salt : 128bit, session random number anti-collision; freeze root : Final audit fingerprint.

[0249] Merge the three roots by salted hash, any single root change will cause the freeze root to mismatch, and the salt value avoids cross-project fingerprint reuse attacks, step 601 integrates write silence detection, evolution list and chain sealing into one, realizes continuous chain of moment capture-item sealing-fingerprint locking, and provides a solid baseline for asynchronous cleaning.

[0250] Evolution freeze scheduler merges the three roots by salted hash, any single root change will cause the freeze root to mismatch, and the salt value avoids cross-project fingerprint reuse attacks, step 601 integrates write silence detection, evolution list and chain sealing into one, realizes continuous chain of moment capture-item sealing-fingerprint locking, and provides a solid baseline for asynchronous cleaning. Accurately capture I / O convergence time, and then seal the snapshot root , repair log root and evolution issuing root into a unique freeze fingerprint with a single chain sealing function; this three-root merging strategy integrates data, repair and historical operations inseparably, completely avoiding the governance vacuum caused by the separation of snapshot and operation log into two chains in traditional archiving process; at the same time, freeze action is only triggered under silent conditions, and does not compete with tail batch gradient write for file handle, eliminating the risk of metadata deadlock common in archiving period.

[0251] Step 602, after the freeze root is released, the background scheduling performs the three-stage process of redundant slice recovery-cache release-version table revision to retain threshold as a criterion to efficiently release space and lock the audit chain.

[0252] More than 90% of redundant slices after freezing have no value for online training; therefore, through the redundant recovery strategy only a subset of cross-node hash slices is retained; thus, the object storage bill is significantly compressed; then, through the cache recovery scheduling vector control the NVMe asynchronous deletion rate to avoid wear peak; and then, after the slice deletion is completed, revise the object storage version table and change the items to the cleanup root Formal addition to the frozen section, ensure the log self-consistent; the final system capacity back to the baseline cost, audit chain still single root can be verified.

[0253] Performing redundancy recovery strategy Wherein the definition of retention probability :

[0254]

[0255] Wherein: the minimum number of data slices : Step two definition, recoverable threshold; slice index : The serial number in the slice set; retention threshold : , the higher the more aggressive cleaning; retention probability : Slice The chance of being retained.

[0256] Ensure at least Data slices must be left, check slices are deleted with exponential decay probability; strategy allows both error and fast space saving. To control the NVMe life cycle, the recovery scheduler is used to:

[0257]

[0258] Adjust the erase rate, dynamically limit the speed to prevent cleaning from triggering SSD write amplification, and prolong the life of the medium:

[0259]

[0260] Wherein: I / O write rate at time The rate changes with the degree of wear of the device;

[0261] Base rate : Recommended IOPS out of the factory; single-day write volume : Today's erase volume, GB; write volume upper limit : Daily health upper limit; wear coefficient : , negative feedback strength; erase depth : The number of blocks touched by a single Trim.

[0262] After deleting the slice, the object storage side updates the version record:

[0263]

[0264] , indicating the updated version index set, containing all indexes retained after the deletion operation;

[0265] , represents the original version index set, containing all initial version indexes;

[0266] , represents the deleted index set, containing the indexes of deleted versions; , represents the set difference operator;

[0267] and calculate:

[0268]

[0269] , represents the value obtained by performing a hash operation on the updated version index set ;

[0270] , represents the hash function, used to encrypt or compress the input data (such as the version index set) to generate a fixed-length output value;

[0271] : represents sorting the indexes in the set. After sorting, input into the hash function for hash calculation to generate the corresponding hash value;

[0272] Finally, write:

[0273]

[0274] to the frozen section + clean patch joint root;

[0275] wherein, represents the frozen version hash value, represents the hash value obtained in the second formula, representing the updated version index, key : Base64 number-time stamp string; clean random salt : 128 bits.

[0276] Audit final root : Unique verification fingerprint after sealing + cleaning.

[0277] Incorporate the clean-up action into the audit chain in the form of a joint root, ensuring that the deletion operation can be reproduced historically. Step 602 first prunes redundancy with a retention probability strategy, then uses write amplification negative feedback to smooth erasing, and finally adds a clean root on the version table level to realize the post-processing closed loop of deletion also being verifiable.

[0278] Through the frozen vector - Evolution list - Sealing function Three-section structure, accurately captures the write silent window at the moment of training end, and adds a snapshot root , repair log root and evolution signing root compressed into single frozen root , from this data view enters read-only-auditable state. Subsequent step 602 with redundancy recovery strategy - cache recovery vector - clean root three-stage flow, safely remove over-provisioning check pieces and link the result fingerprint to the frozen root as the audit final root . The output frozen root of the former is used as the input of the cleaned fingerprint, and the audit final root generated by the latter becomes the final node of the evolution record chain, ensuring that it can still be verified after cleaning.

[0279] recovery phase through exponential decay redundancy recovery strategy only keep the minimum recoverable pieces and a few check pieces, both to maintain the disaster-level recovery bottom line and to immediately reduce the object storage bill to a single-digit percentage of the active period; at the same time, the cache recovery scheduling vector smooth Trim rate according to real-time write volume feedback, cut off the long tail of SSD daily write amplification, extend the life of the medium without sacrificing cleaning speed. The clean root generated after cleaning is merged again with the frozen root to become the audit final root , external supervision only needs to check this root to verify what was stored at the time and what was deleted afterwards, achieving a closed loop of deletion that can be verified.

[0280] Unlike the industry's common backup and deletion or periodic archiving + manual signing, the frozen-cleaning-conjoined root mode of this solution combines data retention, redundancy recovery, and traceable audit into a one-time atomic operation: on the one hand, the retention probability precise control of costs makes object storage expenses decline with model life cycle stages; on the other hand, write amplification negative feedback extends the service life of SSDs, avoiding media avalanches caused by cold storage to hot deletion; more importantly, the audit final root allows any future review to verify what was stored, what was deleted, and when it was deleted with only a single root. From the moment the multi-source perception flow sequence number was born, to the audit final root is permanently sealed, data has gone through a self-consistent, reliable, economical, and verifiable closed loop, laying a clean and sustainable storage foundation for the next round of learning mode.

[0281] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0282] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0283] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0284] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0285] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data storage method for artificial intelligence learning mode, characterized in that: Comprising, The edge high-speed absorption layer assigns monotonic write sequence numbers to multi-source data streams and merges them into sample blocks in real time, providing reliable timing for subsequent erasure coding and reconciliation comparison and ensuring fault replayability; Based on the write sequence numbers, asynchronous erasure coding is performed on the sample blocks to generate cross-node redundant slices and calculate Merkle roots, which are written into the local cache for integrity verification and differential scanning; The edge side sequentially pushes the slices and Merkle root object storage through the bandwidth-aware replication pipeline according to the write sequence numbers, and calls the time travel interface to solidify the incremental snapshot and record the rollback marker; The cloud training scheduler listens to snapshot events to trigger differential scanning, compares write sequence numbers, and if a gap is detected, reconstructs the missing section using the redundant slices and refreshes the time travel index to maintain consistency; wherein the write sequence number chain is generated by a two-segment write sequence number function determined by the logical clock frequency and the carry window size, which keeps the write sequence number globally monotonic and increasing and avoids backtracking by window folding before the counter overflows in a multi-threaded environment; And it is bound to the block merging cache to guide the windowed splicing of multi-source aware streams, and the block merging cache capacity is limited and dequeued in the order of the write sequence numbers; When replaying the training process, a Merkle proof is generated for each sample to verify its integrity. If the verification fails, the damaged sample is marked and repaired using prediction consistency interpolation, and the repair event is recorded; After the training is completed, the write silent window freezes the snapshot to generate the evolution record and lock the frozen root, and then asynchronously deletes the redundant slices to release the cache space and revise the object storage version table to generate the audit final root; After generating the sample blocks, an asynchronous erasure coding scheduler is called, which calculates the priority based on the merging block arrival interval and historical missing block rate and writes it into a preemptible priority queue, and the block with the highest priority is input into the erasure coder; The slice layer appends a timestamp index and a Merkle root digest and writes them into the local cache in a key-value manner to ensure atomic consistency of the block and metadata, and the key-value pair is stored in the reserved transaction area of the high-speed solid-state cache; The asynchronous erasure coding uses a Vandermonde matrix with adjustable redundancy ratio to dynamically adjust the number of data slices and check slices and distribute the generated slices to multiple nodes according to exponential probability using a bandwidth-aware slice distribution algorithm; The asynchronous erasure coding scheduler sorts in real time based on arrival interval, missing block rate, and link margin to ensure that data slices are written to high-bandwidth nodes first and check slices are distributed flexibly according to load, and the multiple nodes include edge nodes and center nodes.

2. The data storage method for artificial intelligence learning mode according to claim 1, characterized in that: After erasure coding is completed, a block-level root is calculated through a hierarchical Merkle tree to generate a sub-root level index, and the write sequence number and timestamp index are spliced to write the slices and Merkle root atomically into the local solid-state cache as key-value pairs; The key-value is a unique sequence number timestamp string for subsequent differential scanning and lake warehouse time travel positioning, the solid-state cache uses non-volatile storage and supports key direct addressing, and provides block root index for subsequent differential scanning.

3. The data storage method for artificial intelligence learning mode according to claim 2, characterized in that: The replication pipeline adjusts the sending rate in real time according to the link utilization, buffer depth and available bandwidth through an adaptive rate control law, and the pipelined acknowledgment window limits the acknowledgment order to the maximum number of in-flight pieces; Any out-of-order acknowledgment is only slid as a whole after the minimum unacknowledged sequence number is acknowledged, ensuring that the push order strictly coincides with the write sequence number and eliminating reverse coverage.

4. The data storage method for artificial intelligence learning mode according to claim 3, characterized in that: When calling the lake warehouse time travel interface to solidify the incremental snapshot, the number of snapshot segments is determined by a square root balancing formula according to the write sequence number span and the version span; A traceable marker containing the block root, version boundary and cyclic redundancy check code is inserted at the end of each segment, and a segment-level hash is generated, so that the segment integrity can be determined by directly comparing the marker without directory traversal, and the segment-level hash result is written into the incremental snapshot metadata area.

5. The data storage method for artificial intelligence learning mode according to claim 4, characterized in that: The differential scanner first calculates the segment-level hash value by comparing the traceable marker with the Merkel root at the end of the segment, and if the hash values do not match, a binary index structure containing a Bloom filter and an eight-sequence number skip list is established to locate the write sequence number gap within two memory accesses; And according to the gap length and time aging coefficient, a reconstruction priority queue is calculated for the reconstruction engine to call, and the gap priority queue is updated in real time in memory.

6. The data storage method for artificial intelligence learning mode according to claim 5, characterized in that: Fast reconstruction uses slice positioning vectors to pull the minimum data slice set that meets the recovery condition at one time, schedules tensor cores to perform inverse Vandermonde matrix operations to recover missing blocks, and only recalculates the affected path nodes of the global Merkel tree through an incremental grafting algorithm; At the same time, the new block key value is appended to the time travel index to maintain query continuity and consistency with the snapshot, and the query continuity involves row consistency of the time travel index.

7. The data storage method for artificial intelligence learning mode according to claim 6, characterized in that: Online verification splices the parent hash by the central processing unit after the tensor core calculates the leaf hash, and the ring buffer manages the proof path with a fixed one-kilobyte slice length; if the verification fails, a damage marker containing the write sequence number, timestamp and first-layer mismatched hash is generated and written into the repair queue, the main training thread ignores the damaged samples by masking and batch, maintains the tensor shape, and the damage marker is stored in the lock-free ring buffer shared memory segment.

8. The data storage method for artificial intelligence learning mode according to claim 7, characterized in that: The interpolation repair algorithm selects the temporally nearest neighbor samples and the model sliding mean within the same batch to generate a replacement tensor, and monitors the gradient disturbance rate in real time; if it exceeds the preset stability threshold, it automatically increases the mean weight and re-interpolates, and writes the repair entry containing the write sequence number, replacement tensor digest and gradient disturbance into the repair log and updates the repair log root, which is synchronized to the upper nodes of the global Merkel tree.

9. The data storage method for artificial intelligence learning mode according to claim 8, characterized in that: By monitoring the three indicators of active write delay, buffer residual depth and refresh completion rate, a freeze vector is formed. When the silence threshold and buffer threshold are met and the refresh completion rate is 100%, a sorted evolution list by write sequence number is generated to calculate the evolution signing root. The snapshot root, repair log root and evolution signing root are salted and hashed to generate the freeze root and broadcast the lock write permission. The evolution list is safely written to the immutable object storage bucket.

10. The data storage method for artificial intelligence learning mode according to claim 9, characterized in that: After generating the freeze root, data pieces are retained according to a retention probability function, and exponential decay deletion is performed on the check pieces. During the deletion process, the write amplification negative feedback scheduling control is used to control the erase-write rate to prolong the service life of the solid-state medium; After completion, the object storage version table is revised to generate a cleaning root, which is jointly hashed with the freeze root to obtain an audit final root. The audit final root is written to the metadata index of the freeze segment.

Images