Cross-network storage domain data updating method based on sparse matrix and storage system

Abstract

The invention discloses a cross-network storage domain data updating method based on a sparse matrix and a storage system, and relates to the field of distributed storage, the method is applied to the distributed storage system adopting erasure codes, and in the method, when a data block needs to be updated, the data block is updated locally based on a Lasso regression algorithm at a storage node; decomposing the update increment into a sparse linear combination and a residual vector of other data blocks in the node; only transmitting the obtained sparse coefficient vector and the residual vector to a target node storing a corresponding verification block; and the target node uses the pre-cached same-node data and combines the generated matrix to calculate the verification update quantity and complete the update. According to the method, the updating flow is reduced to a constant level from the traditional O (m), the calculation localization is realized, and the system throughput and the expandability in a high-frequency updating scene are remarkably improved on the premise of maintaining the fault-tolerant capability of the erasure code.

Description

Technical Field

[0001] This invention relates to the field of distributed storage, and more specifically, to a data update method and storage system based on sparse matrices across network storage domains. Background Technology

[0002] In distributed storage systems, erasure coding technology is widely used due to its high storage utilization, but low data update efficiency remains a major bottleneck. Current update technologies mainly include the following: traditional full update methods, where the update overhead is proportional to the number of parity blocks, leading to excessive data transmission burden across network storage domains; Delta update technology reduces transmission volume by transmitting only incremental data, but still requires reading all old parity blocks, resulting in significant network overhead; Local Repair Code (LRC) limits the update impact to a constant range by introducing local parity groups, reducing the impact of partial updates but sacrificing the maximum distance separability property, thus reducing storage efficiency and fault tolerance; log structure update technology uses append-only writing and delayed merging strategies, avoiding the overhead of in-place updates but introducing new problems such as write amplification and garbage collection. While these methods improve update efficiency to some extent, they still fail to achieve low-overhead and highly scalable update capabilities across nodes while maintaining strong fault tolerance, especially lacking effective solutions for the problem of excessive data traffic across network storage domains.

[0003] Erasure codes have been widely replaced by traditional copying mechanisms due to their high storage efficiency (e.g., (n,k) codes only require n / k times redundancy to tolerate nk node failures). Classical MDS (Maximum Distance Separable) codes (such as RS codes) can tolerate any m=nk node failures, but their low data update efficiency has become a key bottleneck restricting system performance. m is the number of parity blocks; n is the number of data blocks; and k is the number of parity blocks.

[0004] Traditional erasure coding relies on sparse moments for updates, which presents a bottleneck: Taking Rees-Solomon codes as an example, their generator matrix can be represented as follows: ( It is the identity matrix. for (Dense parity-check matrix). When a single data block When an update is needed, because The density (high proportion of non-zero elements). The update will be distributed to all A check block (because check block calculations are designed as a linear combination of all data blocks). Update traffic in network transmission can reach [amount missing]. The block size is determined by the network topology and the location of check blocks and data blocks. For frequent update scenarios (such as small file writes and metadata modifications), network overhead is... The rapid increase in latency severely reduces system throughput and increases latency.

[0005] Limitations of existing optimization schemes: Delta update technology: Delta update technology is a highly efficient update strategy widely used in software distribution, data synchronization, and storage system optimization. Its core idea is to transmit or process only the changed data portion (i.e., the increment), rather than the entire data, thus significantly reducing network traffic and computing resource consumption. In erasure coding storage systems, even when only a single data block is updated (…),… The network throughput still needs to read all old check blocks to calculate the new check value, resulting in a continued high network throughput. In scenarios with frequent data updates, Delta merging overhead may be higher than full write overhead.

[0006] Locally Repairable Codes (LRC): LRC codes introduce local check groups, which require internal check groups when a data block becomes invalid. Individual block repair is required. However, if the updated data block belongs to multiple local check groups, multiple local check blocks and global check blocks still need to be updated. In classic LRC, an update of a single data block affects an average of 2 to 3 check blocks, resulting in constant overhead and sacrificing the MDS property.

[0007] Log structure: This method writes the updated data to a new location, delaying the merge and avoiding in-place updates. However, this increases the overhead of write amplification and garbage collection. Furthermore, it doesn't fundamentally solve the problem; the merge operation still requires costly checksum updates.

[0008] Therefore, existing technologies cannot achieve low-overhead, constant-level data updates across network storage domains while maintaining the strong fault tolerance of erasure coding. Summary of the Invention

[0009] The present invention aims to address the deficiencies in the aforementioned background technology. Its core objective is to provide a data update method and storage system based on sparse matrices across network storage domains. While maintaining the original fault tolerance and storage efficiency of erasure coding, it breaks through the bottleneck that cross-node network traffic during data updates is proportional to the system size m, achieving constant-level update overhead, thereby significantly improving the performance and scalability of distributed storage systems in high-frequency update scenarios.

[0010] To achieve the above-mentioned objectives, this invention provides a data update method for cross-network storage domains based on sparse matrices, applicable to distributed storage systems employing erasure coding. The method includes: Step 1: Preprocess and divide the raw data into blocks to obtain... k Data blocks ; Encoded by erasure coding, by the k Data blocks generated One check block, k and m All are integers greater than 1; the data blocks and check blocks are distributed and stored across multiple storage nodes; Step 2, when data blocks It needs to be updated to its new value. At that time, calculate data blocks Update increment , where 1≤ i ≤ k ; Step 3: Based on the update increment The sparse coefficient vector is obtained by using the Lasso regression algorithm. Based on incremental updates Sparse coefficient vector Calculate and obtain the residual vector ; Step 4: Convert the sparse coefficient vector and the residual vector Transmit to storage of the data block d i The other storage node of the corresponding check block is used as the target node; Step 5: In the target node, according to the sparse coefficient vector and the residual vector Calculate the verification update amount Complete the update of the verification block.

[0011] This invention addresses the issue of excessive cross-network storage domain update traffic in erasure coding technology within distributed storage systems. Traditional erasure coding (such as Reed-Solomon) suffers from high I / O overhead due to its dense parity-check matrix, requiring simultaneous modification of the entire parity block for a single data block update (resulting in significant cross-node I / O overhead). Addressing the shortcomings of existing optimization schemes in balancing sparsity and fault tolerance, this invention proposes a cross-node update optimization scheme based on sparse linear approximation and residual correction. This invention is the first to introduce sparse optimization theory (Lasso) into the field of erasure coding updates, mathematically constructing a mechanism to forcibly decouple the update dependencies of data blocks and check blocks, thus overcoming cross-node traffic bottlenecks while maintaining fault tolerance. Its constant-level overhead (independent of system size m) provides an engineering-ready erasure coding solution for distributed hot data storage, driving the evolution of storage architecture towards higher efficiency and lower cost.

[0012] This invention is the first to introduce Lasso regression (sparse optimization theory) into the field of erasure coding updates. Through mathematical construction, it decouples and localizes the global update calculation, thereby reducing cross-network data transmission at the source.

[0013] This method updates the data block incrementally. By solving an L1 regularized (Lasso) linear regression problem, using other data blocks (matrices) within the same node... B A sparse linear approximation is performed. The sparse coefficient vector is obtained by solving this problem. and residual vector The update operation is transformed into simply transmitting these two small vectors to the target node, and then utilizing the matrix pre-stored in the target node. B The final verification calculation is completed. This solves the O(m) network transmission bottleneck in traditional methods, where updates must be propagated to all verification blocks due to the dense generator matrix.

[0014] Preferably, the sparse coefficient vector The calculation method is as follows: ; in, B For the data block d i The matrix formed by other data blocks within the same storage node. λ >0 is the regularization parameter; For, update increment The calculation method is as follows: residual vector The calculation method is as follows: .

[0015] Preferably, the verification update volume The calculation method is as follows: ; in, To verify the increment, G The generator matrix of erasure coding, B For the data blockd i The matrix formed by other data blocks within the same storage node. For regularization parameters, It is a standard unit column vector.

[0016] Among them, the verification update volume It consists of two parts: one is the residual vector. By generating matrices The i List The direct contribution generated; secondly, the sparse approximation part. Through the generator matrix of complete erasure coding The contribution made is to ensure the mathematical correctness and completeness of the verification update after introducing sparse approximation, making the results of local computation equivalent to the theoretical results of global update.

[0017] Preferred, The calculation method is as follows: , To generate the matrix The i List, The dimension of the matrix The numbers in the matrix are all in the real number field.

[0018] Preferably, preprocessing of the raw data includes: Perform anomaly detection on the raw data based on a sliding window, and identify and mark data anomaly windows; The raw data is cleaned in three stages, including feature extraction, priority sorting, and deduplication and merging. Unstructured data in the original data is selected and standardized in format, and then converted into a unified structured representation.

[0019] Before encoding and updating, the original unstructured data undergoes standardization cleaning, anomaly detection, feature selection, and format conversion to form high-quality, well-organized data blocks. This addresses the issue that noise, heterogeneity, and irregularity in the original data can affect the effectiveness and efficiency of subsequent block segmentation, encoding, and sparse decomposition. Improving data quality lays the foundation for subsequent erasure coding and efficient sparse representation, thereby enhancing the overall robustness of the method and the final compression and update efficiency.

[0020] Preferably, the verification data of the target node is finally updated using the following formula: ; in, For the updated verification data, This is the verification data before the update. For regularization parameters, G The generator matrix of erasure coding, B For the data block The matrix formed by other data blocks within the same storage node. For verification increment.

[0021] Preferably, all check blocks are stored on different storage nodes. During data layout, all check blocks are distributed across different physical nodes. This avoids a single node becoming an I / O hotspot due to storing too many check blocks, and also prevents the loss of too many check blocks due to a single node failure, which would affect repair efficiency and improve system load balancing and reliability.

[0022] Preferably, in step four, only the sparse coefficient vector is transmitted. The non-zero elements and their corresponding index information in the array.

[0023] Among them, the sparse coefficient vector generated by Lasso regression The characteristic of sparseness (most elements are zero) means that only non-zero values ​​and their position indices are sent during transmission. This can further compress the size of data packets that need to be transmitted over the network. On the basis of the previous technical solutions that have reduced overhead, the number of bits actually transmitted over the network is reduced again, further optimizing bandwidth utilization.

[0024] Preferably, the residual vector Compress the data before transmission.

[0025] Transmitting residual vector Previously, compression algorithms (such as entropy coding) were used to compress the residual vector. However, the residual vector may still contain some information, making direct transmission less than optimal. Further reductions in the amount of data transmitted across the network and improvements in update efficiency are particularly effective for cases with large residuals.

[0026] Preferably, the erasure code is a Reed-Solomon code, and it maintains the maximum distance separability attribute.

[0027] Preferably, the Lasso regression calculation and the residual vector calculation in step three are both performed on the data block. d i The process is completed locally on the storage node.

[0028] Specifically, the computationally intensive Lasso regression solution and residual calculation processes are confined to the source storage node initiating the update. This avoids the additional network round-trip latency and computational overhead caused by data collection and collaborative computation across multiple nodes for sparse decomposition. It also shortens the response time of update requests and reduces the network bandwidth consumption of the update process.

[0029] The present invention also provides a distributed storage system for executing the aforementioned sparse matrix-based cross-network storage domain data update method, comprising: Multiple storage nodes are used to store data blocks and check blocks; Meta nodes are used to coordinate data distribution and update scheduling; The preprocessing module is used to perform data cleaning, standardization, and chunking; The update processing module is deployed in the storage node and is used to perform Lasso sparse decomposition, residual calculation and verification update.

[0030] One or more technical solutions provided by this invention have at least the following technical effects or advantages: This invention, while maintaining the original fault tolerance and storage efficiency of erasure coding, breaks through the bottleneck that cross-node network traffic during data updates is proportional to the system size m, achieving constant-level update overhead, thereby significantly improving the performance and scalability of distributed storage systems in high-frequency update scenarios.

[0031] Constant-level network overhead: Transmission volume depends only on the sparse coefficient vector The non-zero terms and residuals are related, and are independent of the total number of check blocks m in the system, fundamentally reducing the network overhead from O(m) to the constant level.

[0032] Maintaining fault tolerance: The entire calculation process is based on the generator matrix of erasure codes. G The derivation shows that the erasure coding structure remains unchanged, thus fully preserving its MDS properties and fault tolerance.

[0033] Computational localization: The most time-consuming Lasso optimization computation is performed only locally on the source node, avoiding the overhead of cross-node collaborative computation. Attached Figure Description

[0034] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention. Figure 1 This is a flowchart illustrating a data update method across network storage domains based on sparse matrices. Figure 2 A schematic diagram of the node deployment structure; Figure 3 Flowchart for data update. Detailed Implementation

[0035] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0036] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0037] Those skilled in the art should understand that, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention.

[0038] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0039] Example 1; Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a data update method across network storage domains based on sparse matrices. Embodiment 1 of the present invention provides a data update method across network storage domains based on sparse matrices, applied to a distributed storage system employing erasure coding. The method includes: Step 1: Raw Data Preprocessing and Blocking: The raw data is preprocessed and divided into blocks to obtain... k Data blocks ; Encoded by erasure coding, by the k Data blocks generated m One check block, k and m All are integers greater than 1; the data blocks and check blocks are distributed and stored across multiple storage nodes; Step 2: Calculate the update increment: When the data block d i It needs to be updated to its new value. At that time, calculate data blocks d i Update increment , where 1≤ i ≤ k ; Step 3: Calculate the sparse coefficient vector and residual vector: based on the update increment. The sparse coefficient vector is obtained by using the Lasso regression algorithm. Based on incremental updates Sparse coefficient vector Calculate and obtain the residual vector ; Step 4: Transmit the sparse coefficient vector and residual vector: Transmit the sparse coefficient vector... and the residual vector Transmit to storage of the data block The other storage node of the corresponding check block is used as the target node; Step 5: Update: In the target node, according to the sparse coefficient vector and the residual vector Calculate the verification update amount Complete the update of the verification block.

[0040] Specific embodiments of the present invention include: Node deployment process: Please refer to Figure 2 , Figure 2 A schematic diagram of the node deployment structure. Figure 2 Rack1, Rack2, and Rack3 are three different racks. d1 to di+1 represent data blocks, and p1 to pj represent parity blocks. In a distributed storage system, to achieve efficient deployment of erasure coding, the original data needs to be adapted to the storage constraints of the network nodes. This invention adopts a layered processing architecture, uniformly coordinated by a central meta node. The specific process is as follows: Data preprocessing: The data is preprocessed using a four-stage pipeline architecture to standardize the raw unstructured data, laying the foundation for subsequent feature extraction and intelligent classification.

[0041] First, the raw data is cleaned and deduplicated to eliminate noise interference. Second, format normalization is performed to unify encoding and timestamp standards. Next, the data type and key fields are identified through the semantic parsing module. Finally, metadata annotation is completed and index information is built to ensure that data blocks can be efficiently scheduled and located. The entire process is monitored in real time by the control bus to ensure processing consistency.

[0042] Anomaly detection based on sliding window: The records in the unstructured cloud dataset are sorted according to corresponding keywords. A fixed-size detection window (usually 512 records) is set, and the window is moved sequentially to detect the matching degree between the data records in the window and the keywords. The matching degree is calculated using the following formula: ; in, For matching degree, when the matching degree is lower than the threshold (85%), the window is marked as an abnormal window.

[0043] Three-stage cleaning process: Key feature attributes are extracted from the raw data, including metadata such as timestamps, data size, data type, and access frequency. The data is prioritized according to the importance weight of each feature value, ensuring that high-quality data is processed first. The cleaned data is then reorganized and merged to eliminate duplicate records, forming a well-structured dataset.

[0044] Unstructured cloud data selection: To reduce processing complexity, a representative subset of data is selected from the original dataset. A selection model is established based on the criterion that the selected data within the dataset share the same features as the feature values. ; in, To select a standard value, representing the overall consistency between the entire data subset and the target feature, the value range is usually [0, 1], with higher values ​​indicating better consistency. Let be the actual value of the i-th feature, derived from the data object to be evaluated; n be the number of features, representing the total number of features extracted from the data for comparison; and i be the feature index, indicating the i-th feature currently being compared. This is the feature similarity function. The target feature value is defined by an improved random sampling method that ensures balanced selection of all data types, avoiding selection bias.

[0045] Transformation of unstructured data: Heterogeneous data is converted into a unified format to facilitate subsequent processing and analysis. According to user settings, selected structured data is converted into XML format, establishing a standardized data description framework. Logarithmic transformations are applied to numerical functions to perform XML structuring on semi-structured data, and semantic annotation and classification of text data are performed based on ontology classes.

[0046] Divide the standardized data into blocks: In distributed storage systems, to achieve efficient deployment of erasure coding, the processed raw data needs to be adapted to the storage constraints of network storage nodes. This invention adopts a hierarchical intelligent processing architecture, coordinated by a central meta-node. The specific process is as follows: Let the original data to be stored be... The system contains N storage nodes, and each node has an available storage space of [missing information]. (Unit: bytes). Cut into Data blocks Each block is Dimensional vector: ; in, For data blocks Each number in, w For the bit width of a finite field, This represents a finite field (GaloisField) used for subsequent erasure coding.

[0047] RS code encoding: Encoding is performed using RS code, and the generated matrix G consists of the identity matrix. and Vandermonde matrix constitute: ; in, It is the identity matrix. For the numbers in the Vandermonde matrix, For the check block vector, For a finite field The distinct non-zero elements in the matrix ensure that the matrix is ​​full rank (MDS property). Check block vector Through matrix multiplication: ; in, To verify the numbers in the block vector, T is the transpose operator, and d is the data block vector. The number is the transpose of the data block vector; That is, each check block for: ; Data is distributed to storage nodes: Meta nodes maintain network topology tables This involves mapping storage nodes to physical nodes. The core principle of parity block distribution is to distribute all parity blocks as widely as possible across different network nodes. This ensures that a single node failure will not result in the loss of all parity blocks and also prevents some nodes from becoming I / O hotspots due to storing too many parity blocks.

[0048] Lasso sparse decomposition: Please refer to Figure 3 , Figure 3 The data update flowchart illustrates how, when a data block in a distributed storage system needs updating, this invention proposes an innovative sparse decomposition method based on Lasso regression. This method transforms the traditionally globally-involved update operation into an efficient local computation problem, significantly reducing cross-node data transmission overhead. Specifically, let the data block to be updated be... Its new value is The update increment can then be expressed as .in, Figure 3 Data matrix B in the same rack is a local material library, and data matrix B needs to be retrieved locally for subsequent calculations.

[0049] In terms of system architecture design, this invention fully utilizes the locality characteristic of data center network topology. Each storage node maintains the same data matrix internally. It is listed as other data blocks on the same storage node. This design allows the system to prioritize the use of data resources within the same network node when updating data.

[0050] Regarding cross-node traffic optimization, the core idea of ​​this invention is to approximate the update increment using sparse linear representation with as few data blocks as possible from the same storage node. In general, we hope to find a sparse linear combination. ,in It is a sparse coefficient vector. This representation can significantly reduce the amount of data that needs to be transmitted across network nodes, because only the residual part that cannot be represented by the same node needs to be transmitted.

[0051] This invention employs the Lasso regression algorithm to achieve this goal, which can automatically identify and select pairs of representations. The data block that contributes the most to the same network node is selected, while data blocks with smaller contributions are ignored. This characteristic stems from the L1 regularization term of Lasso regression, which can produce sparseness. The specific optimization problem is expressed as: ; in This is a regularization parameter used to control the sparsity of the solutions. In practical applications, The initial value is usually set to 1, but it can be dynamically adjusted according to network conditions and accuracy requirements. This is the regularization parameter, usually set to 1. By solving this optimization problem, we can obtain a sparse coefficient vector. The non-zero elements correspond to the key network node data blocks that need to be updated.

[0052] Residual processing and verification updates: After completing the Lasso sparse decomposition, the system accurately calculates the residual vector, a crucial step in the entire update process. The system needs to process the residuals and coefficient verification increments to further optimize the verification update efficiency. The specific process is as follows: Calculation of residual and sparsity check increments: Let the residuals obtained after Lasso regression be: ; Simultaneously calculate the verification increment: ; in, It is the generating matrix The Columns represent the current data block. Contribution weights to all check blocks.

[0053] Assume the target network node has already cached the data matrix of the same node. Only the residuals need to be considered. and sparse coefficient vector Simply transmit it to the target node.

[0054] pass This completes the calculation of the verification update increment. The specific derivation process is as follows: ; The final verification data of the target network node is updated using the following formula: ; In this invention, the data update amount is innovatively decomposed into two parts by Lasso regression: a sparse linear combination of data from the same network storage node and the residual after compression, thus avoiding the update amount required by traditional RS code to transmit complete data blocks.

[0055] This invention transmits only non-zero coefficients, which greatly reduces the amount of data transmitted, optimizes bandwidth usage across network storage domains, and improves update efficiency.

[0056] In this invention, the amount of data that needs to be transmitted is further reduced by compressing the residuals generated during the update, thereby optimizing the storage efficiency of the system.

[0057] Compared to traditional RS codes, this invention fundamentally breaks through the transmission bottleneck in large-scale distributed storage systems through a sparse decomposition mechanism, and greatly improves performance, especially in high-frequency update scenarios.

[0058] This invention, through its innovative distributed computing architecture, completely confines Lasso regression and residual calculation within the source network storage node, thus avoiding cross-node collaborative computation.

[0059] The present invention completely eliminates the network latency and computational overhead caused by cross-network storage domain computation in traditional solutions through a localized computing strategy, and significantly reduces the pressure on network transmission.

[0060] In this invention, all sensitive data is calculated within the source network storage node, avoiding the risk of data leakage during cross-node transmission and significantly improving the system's security and privacy protection capabilities.

[0061] The key point of this invention is that, through an innovative Lasso sparse decomposition mechanism and a localized computing architecture, the data block update volume is decomposed into a sparse linear combination of data from the same network storage nodes and residuals, requiring only the transmission of non-zero coefficients and residuals. All computations are performed entirely within the source node, avoiding cross-node coordination overhead.

[0062] This invention is the first to apply Lasso regression to erasure coding data updates. By transforming global updates into local computations through sparse decomposition, it reduces cross-network node transmission traffic. The computation time is reduced from the traditional O(m) to constant level, making it suitable for high-frequency update scenarios.

[0063] During updates, the update amount is decomposed into a sparse linear combination of data from the same node plus residuals. Only non-zero sparsity and residuals are transmitted, rather than the entire data block.

[0064] Both Lasso regression and residual calculation are performed within the source network node, avoiding network latency and computational overhead caused by cross-node collaboration. Localized computation reduces network latency and improves response speed.

[0065] Sparse updates are achieved while maintaining erasure coding MDS properties, without affecting fault tolerance.

[0066] Data preprocessing includes window-based anomaly detection, three-stage clarification, and standardized processing steps for unstructured data selection and variation.

[0067] The verification blocks should be distributed across different network nodes as much as possible to avoid single-node failures and I / O hotspots.

[0068] Example 2; Based on Embodiment 1, Embodiment 2 of the present invention will be described in conjunction with specific examples: In a distributed storage system, there are two network nodes using RS(4,2) erasure coding. Each node stores two data blocks and one parity block. Now, one data block in node 1 needs to be updated. We use the method of Example 1 to calculate and transmit the parity block. The following is a case description.

[0069] (1) The generating matrix is initial value .

[0070] The calculated check values ​​are respectively .in as well as The data blocks are stored in node 1, while the remaining data blocks and check blocks are stored in node 2.

[0071] (2) Update request ,but The verification data in node 1 uses Simply update.

[0072] (3) Sparse decomposition of the source node. Data matrix of the same node. It has already been cached in node 2.

[0073] set up Solve for sparsity coefficients .get .

[0074] Calculate residuals .

[0075] (4) The residual and sparsity coefficient The data is transmitted to node 2, where further calculations are performed.

[0076] Calculate the verification increment: ; Calculate the updated verification data: ; Finally, the verification data of node 2 was updated to .

[0077] Example 3; Based on Embodiment 1, Embodiment 3 of the present invention provides a distributed storage system for executing the cross-network storage domain data update method based on sparse matrices described in Embodiment 1, comprising: Multiple storage nodes are used to store data blocks and check blocks; Meta nodes are used to coordinate data distribution and update scheduling; The preprocessing module is used to perform data cleaning, standardization, and chunking; The update processing module is deployed in the storage node and is used to perform Lasso sparse decomposition, residual calculation and verification update.

[0078] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0079] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for data update across network storage domains based on sparse matrix, applied to a distributed storage system using erasure code, characterized in that, The method includes: Step one, pre-processing and blocking the original data to obtain k data blocks ; generating m check blocks from the k data blocks through erasure code encoding, k and m are both integers greater than 1; dispersing the data blocks and check blocks in multiple storage nodes; Step two, when data block d i needs to be updated to its new value , the update increment of data block d i is calculated , where 1≤i≤k. Step three, compute the updated increment based on the updated residual vector Solve the Lasso regression algorithm to obtain a sparse coefficient vector , based on the updated increment and the sparse coefficient vector Compute the updated residual vector ; Step four, transmitting the sparse coefficient vector and the residual vector to another storage node storing a check block corresponding to the data block d i as a target node; Step five, in the target node, according to the sparse coefficient vector and the residual vector , calculate the check update amount , complete the update of the check block.

2. The method of claim 1, wherein, the sparse coefficient vector is computed as ； Where B is the data block d i The matrix formed by other data blocks within the same storage node, where λ>0 is the regularization parameter; For regularization parameters, update the increment. The calculation method is as follows: residual vector The calculation method is as follows: .

3. The method of claim 1, wherein, Check update amount The calculation is as follows: ； wherein, is a check increment, G is a generator matrix of the error correction code, B is a matrix of other data blocks in the storage node where the data block d i is a matrix of other data blocks in the storage node where the data block d is a regularization parameter, is a standard unit column vector.

4. The data update method across network storage domains based on sparse matrices according to claim 3, characterized in that, The calculation is: , is the i-th column of the generator matrix G, is the dimension of the matrix .

5. The method of claim 1, wherein, Preprocessing of raw data includes: Perform anomaly detection on the raw data based on a sliding window, and identify and mark data anomaly windows; The raw data is cleaned in three stages, including feature extraction, priority sorting, and deduplication and merging. Unstructured data in the original data is selected and standardized in format, and then converted into a unified structured representation.

6. The method of Claim 1, wherein, The target node's verification data is finally updated using the following formula: ； in, For the updated verification data, This is the verification data before the update. Let G be the regularization parameter, G be the generator matrix of the erasure coding, and B be the value of the data block d. i The matrix formed by other data blocks within the same storage node. For verification increment.

7. The method of claim 1, wherein, All check blocks are stored on different storage nodes.

8. The method of Claim 1, wherein, In step four only the non-zero elements of the sparse coefficient vector and their corresponding index information are transmitted In step four only the non-zero elements of the sparse coefficient vector and their corresponding index information are transmitted 9. The method of claim 1, wherein, The residual vector The compression processing is performed before transmission.

10. A distributed storage system for executing the data update method based on a sparse matrix across network storage domains as described in any one of claims 1 to 9, characterized in that, include: Multiple storage nodes are used to store data blocks and check blocks; Meta nodes are used to coordinate data distribution and update scheduling; The preprocessing module is used to perform data cleaning, standardization, and chunking. The update processing module is deployed in the storage node and is used to perform Lasso sparse decomposition, residual calculation and verification update.

Images