Distributed storage method based on source code semantic partitioning

Abstract

The invention provides a distributed storage method based on source code semantic partitioning, and particularly relates to the technical field of cloud data distributed storage. The method comprises the steps of performing semantic partitioning on a source code, and segmenting the source code into a plurality of semantic blocks according to dimensions such as functional semantics, an abstract syntax tree structure, author information and version information; generating metadata containing information such as grammar type tags, file paths, line number ranges, author identifiers, version identifiers and access popularity for each semantic block; constructing a weighted directed acyclic graph (DAG) based on the semantic chunks and the dependency relationship thereof; superposing a metadata layer in the DAG structure, and recording information such as function call dependency, inter-block reference relationship and version evolution chain; blocks with relatively high access frequency and close semantics are aggregated into super blocks, the traversal depth is reduced, and meanwhile, hot data and cold data are differentiated for hierarchical storage by adopting a cold and hot data management strategy; and evaluating a parent block aggregation degree through a BDS algorithm, determining a block sorting priority, and optimizing super block boundary division. Compared with the prior art, the method has the advantages that the semantic retrieval efficiency, the incremental updating capability and the distributed query performance of the source code storage system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud data distributed storage, and in particular to a distributed storage method based on source code semantic segmentation. Background Art

[0002] Distributed storage systems (such as IPFS) usually use content addressing to uniquely identify and find data, and index the data by the hash value (CID) generated by its content; however, existing distributed storage systems (such as IPFS) use Merkle The DAG (Directed Acyclic Graph) structure is used to organize and verify data. The identity of each node is calculated from the hash value of its content and child node list. This structure ensures data integrity and immutability, but its organization essentially relies solely on the hash value of the content, without considering the semantic structure of the file or code. In other words, systems such as IPFS divide data into fixed-size or sliding-window-based segments and associate them through hashes. However, these segments do not consider logical boundaries such as functions, classes, and modules in the source code. Traditional text segmentation algorithms (such as fixed-size segmentation and Rabin segmentation) often treat code as sequences of characters or tokens and lack an understanding of source code semantics. For example, existing segmentation can break at any point in a function body, destroying the semantic integrity of functions and classes, making it difficult to locate code blocks based on their actual business meaning during retrieval. In addition, metadata management in distributed storage systems generally focuses on file location or basic attribute information, but lacks additional organization of code semantic information. For large-scale storage, it is also necessary to manage hot and cold data separately to improve performance and save costs. Zhou and Xinhuajing, in "Tiered-Indexing: Optimizing Access Methods," discuss this issue. Statistics in "forSkew" show that the vast majority of data in distributed systems is cold data, and only a small amount of data is frequently accessed. Therefore, managing hot data and cold data separately can effectively improve system performance. Existing technologies have not yet effectively combined the semantic structure of source code and distributed storage performance optimization for unified processing and management.

[0003] The present invention proposes a distributed storage method based on source code semantic segmentation. Through technical means such as semantic code segmentation, construction of weighted DAG structure, super-block aggregation management, and BDS algorithm optimization and aggregation sorting, the method realizes semantic organization, relational indexing and multi-dimensional metadata management of source code in a distributed storage environment. Compared with the existing IPFS storage method that only relies on content hash value to organize data, the present invention demonstrates better code semantic retrieval efficiency, more efficient incremental update and version traceability capabilities. At the same time, through tiered storage of hot and cold data and super-block aggregation management, it significantly optimizes the access performance and storage utilization of large-scale code data in a distributed environment, thereby improving the overall retrieval performance and scalability of the system. Summary of the Invention

[0004] To address the lack of semantic organization in existing content hash-based storage, this paper proposes a distributed storage method based on source code semantic segmentation. By segmenting and organizing source code at a semantic level, the paper aims to build an index layer in the distributed storage system that describes the code structure and relationships, thereby improving the retrieval efficiency and storage utilization of source code data.

[0005] To achieve the above object, the present invention adopts the following technical solutions, including:

[0006] First, the source code of the target application is semantically segmented into multiple semantically independent code blocks based on logical boundaries such as functions, classes, and modules. The source code is divided based on multiple dimensions such as the functional semantics, abstract syntax tree structure, author information, and version information. Compared with traditional character-level or fixed-size segmentation, semantic segmentation ensures that each block remains semantically complete and does not cross function or class definitions, thus better conforming to developers' understanding of code structure.

[0007] Metadata is generated for each semantic code block, including code block identifier, size, hash value, code dependencies, context tags and other information; these metadata record various properties of the code block and the semantic associations between them, facilitating subsequent retrieval and management.

[0008] Then, using the aforementioned semantic code blocks as nodes, a directed acyclic graph (DAG) is constructed, where directed edges between nodes represent dependencies or call relationships between code blocks. Unlike traditional Merkle DAGs, this solution adds weight information to edges or nodes. This weight can be calculated based on the importance of the code block, call frequency, or size, reflecting the "weight" of each node in the overall structure; this is similar to the idea of ​​using a weight mechanism to adjust performance in distributed ledgers.

[0009] After constructing the weighted DAG, the previously generated semantic metadata is associated with the DAG nodes through a metadata overlay. This overlay is essentially a mapping: each DAG node (code block) corresponds to one or more metadata records, combining content data with semantic labels. During retrieval, the query module first locates the matching code block ID based on semantic attributes in the overlay, then navigates to the specific storage block through the DAG structure. This design allows the query process to be independent of the complex underlying hash addressing structure, relying solely on semantic indexing.

[0010] Define a superblock to manage and aggregate metadata or index information for multiple code blocks. Multiple frequently accessed, semantically dense blocks are aggregated into a single physical superblock while preserving the semantic boundaries between blocks, facilitating rapid location and management of large datasets.

[0011] Data is categorized and managed as hot and cold data based on access frequency and timing characteristics. Hot data, frequently accessed, can be cached in high-performance storage first, while cold data, rarely accessed, can be placed in a low-cost storage tier. A tiered management strategy is employed, storing hot data in the memory cache tier and cold data persistently stored on disk or in a low-speed storage tier. This hot-cold tiering approach significantly improves system access performance while conserving storage resources.

[0012] During the aggregation or evaluation phase, the Binary Decision Scheme (BDS) algorithm is used to aggregate and evaluate code blocks, accelerating the structure traversal and matching process. The BDS algorithm reduces real-time computing overhead by pre-calculating some nodes or paths, thereby improving the efficiency of DAG structure operations.

[0013] As described above, the distributed storage method based on source code semantic segmentation of the present invention has the following beneficial effects:

[0014] This invention achieves semantic organization and efficient management at the source code level in a distributed storage system. Incorporating code semantics into the distributed storage index enables semantic-based retrieval and management. Combining DAG structures, metadata layers, and hot and cold tier management techniques improves storage system search efficiency and data management efficiency.

[0015] In general, the method of the present invention has the advantages of both data integrity assurance and semantic retrieval convenience, and improves the availability and performance of distributed storage of code data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The advantages of the above aspects of the present invention will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0017] FIG1 is a flow chart of a distributed storage method based on source code semantic segmentation according to a specific embodiment of the present invention;

[0018] FIG2 is a schematic diagram of a hot and cold data management process according to a specific embodiment of the present invention;

[0019] FIG3 is a schematic diagram of detailed steps of BDS algorithm aggregation evaluation according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Example 1: Please refer to Figure 1 As shown in the figure, this example provides a detailed implementation of a distributed storage method based on source code semantic segmentation. The storage process includes the following steps:

[0022] Semantic chunking: This step takes the target application's source code as input and uses code analysis techniques (such as AST parsing) to identify semantic boundaries. In software source code, function definitions, class declarations, module imports, and other structures typically constitute logically complete semantic units. The semantic chunking module scans the code files, detecting the start and end positions of structures such as functions and classes, and uses these positions as chunking boundaries. For example, a function declaration and its corresponding function body are considered a code block, with each code block containing semantically self-consistent, continuous lines of code. Each semantic code block generated in this way retains the original code logic while becoming a node in the DAG.

[0023] Metadata generation: This step constructs corresponding metadata records for each code block generated by the semantic segmentation module. The metadata usually includes: the unique identifier of the code block (such as a content-based hash value), the block size, the file path to which the code belongs, the function / class name, and the code dependency reference information, such as other functions or modules called. These metadata records are stored in a distributed metadata database or key-value storage. The metadata generation module can also extract key semantic information (such as variable relationships, annotation keywords, etc.) and append it to the metadata layer to enrich the semantics of the code block. The design of the metadata layer is similar to maintaining an "index item" for each code block to provide support for subsequent fast queries.

[0024] Constructing a weighted DAG uses the code blocks derived from semantic segmentation as nodes. Directed edges are constructed based on the semantic dependencies between code blocks (e.g., function calls, module references, inheritance relationships, etc.), forming a directed acyclic graph (DAG). In traditional IPFS Merkle DAGs, each node ID is a hash of its content and child node IDs. The DAG of this invention builds on this by introducing the concept of weights. Specifically, a weight can be assigned to each edge or node to reflect its importance based on the size of the code block, call frequency, or other business metrics. This weighted DAG structure leverages the advantages of content addressing (linking nodes through hashes) to ensure data immutability while also providing semantic "priority" for code blocks through a weighting mechanism. Much like the use of weighting to adjust performance in distributed ledger systems, the weighted mechanism employed in this invention can help the system optimize the processing order or caching strategy of code blocks based on business needs.

[0025] After constructing the weighted DAG, the metadata layer overlay module associates the previously generated semantic metadata with the DAG nodes. Later, when a query is needed, the overlay layer first locates the code block ID that meets the conditions based on the semantic attributes, and then navigates to the specific storage block through the DAG structure. This design allows the query process to be independent of the complex underlying hash addressing structure, simply operating on the semantic index. For example, it is possible to directly query the code block containing a key comment or function name without traversing the entire graph structure.

[0026] Superblock generation: This step defines and generates superblocks to manage large-scale code block collections. Superblocks are similar to superblocks in file systems, storing overall statistical and index information. Specifically, superblocks can contain information such as the current storage system version, the total number of code blocks, and the entry identifiers of each major code module. By regularly updating or dynamically maintaining superblocks, the system can quickly obtain global information, such as the root node of the entire DAG, the total number of blocks, and space usage. This helps speed up system startup, loading, and fault recovery, and also facilitates the development of global caching or replication strategies.

[0027] Hot and cold data management, see Figure 2As shown, this step employs a hot-cold tiered storage strategy to manage semantic code blocks by categorizing them into hot and cold. First, the access frequency and access time slot of each code block are calculated by analyzing access logs or real-time access patterns. These blocks are then marked as hot or cold data based on preset thresholds. Hot data is preferentially cached in high-performance storage media (such as SSDs or memory cache) to reduce access latency. Cold data is moved to a lower-cost persistent storage layer. Utilizing the concept of tiered management (Cache Layer + Persisted Layer), as hot data loses popularity, it can be gradually moved to the cold layer. If cold data suddenly becomes hot, it can be promoted to the cache layer. This dynamic hot-cold swapping improves storage resource utilization while ensuring access performance.

[0028] BDS algorithm aggregate evaluation, see Figure 3 As shown, this step uses the BDS algorithm to accelerate the data aggregation and comparison stage. The BDS algorithm is commonly used in the signature and verification of Merkle tree structures. It reduces the number of real-time traversals by pre-calculating some nodes, thereby reducing overhead. In the present invention, the BDS algorithm can be used to quickly evaluate the similarity or inclusion relationship between multiple code blocks and support aggregation operations based on incremental updates. For example, when adding or modifying a code block, the cumulative weight or check value of the relevant DAG path is quickly updated through the BDS algorithm without having to re-traverse the entire subgraph. This not only improves maintenance efficiency, but also reduces the demand for real-time computing resources of the system.

[0029] If there is a query requirement, the method provides users with a semantic-based retrieval interface. Users can enter query conditions such as function names, module names, or key statements. First, the code block IDs that meet the conditions are retrieved in the overlay metadata layer, and then the corresponding data fragments are obtained through the DAG structure. With the block granularity consistency brought by semantic block segmentation, the query results can be directly located to the complete code block, improving the retrieval accuracy. For more complex queries, the system can trace the reference relationship between blocks along the weighted DAG, and further associate related code blocks to support cross-file or cross-module code association retrieval.

[0030] In summary, this invention, through the collaborative work of the above functional modules, achieves semantic organization and efficient management of source code in distributed storage. This allows the storage system to not only focus on data content integrity but also incorporate code structure and business semantics, effectively addressing the shortcomings of traditional content-addressing methods and improving the retrieval efficiency and user experience of distributed source code storage systems.

[0031] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed storage method based on source code semantic segmentation, characterized in that: The method comprises: S1. Divide the source code into multiple fixed-size semantic blocks based on the source code's functional semantics, abstract syntax tree structure, author information, and version information. S2. Generate metadata containing labels for each segmented semantic block, including but not limited to the syntax type label, file physical path, author identifier, version identifier, and access popularity of the segmented block; S3. Construct a weighted directed acyclic graph (DAG) using the semantic blocks as nodes. The edges of the DAG represent logical associations or dependencies between nodes, and weights are assigned to the edges to reflect the strength of the associations. S4. Add metadata such as function call dependencies, inter-block reference relationships, and version evolution chains to nodes and edges in the DAG; S5: Aggregate multiple semantic blocks with high access frequency and close semantics into a physical super block while preserving the semantic boundaries between blocks for cold data merging strategy; S6. Cold and hot data are distinguished using the BDS aggregation evaluation algorithm to calculate the mean and variance of the parent block aggregation degree of each block, and determine the block sorting priority based on the variance. The priority is directly used as the boundary identification to guide the super block merging.

2. A distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The source code is divided by traversing the abstract syntax tree of the source code, taking the AST subtree as a semantic block, and merging adjacent brother nodes in the AST to generate a semantic block while ensuring the block size limit.

3. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The metadata is generated, and the tags generated for each semantic block include the AST node type of the block, such as loop structure, conditional structure, function definition, etc.

4. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: In the graph constructed by the DAG construction module, nodes represent code semantic blocks, and edges are generated based on the call relationship, data dependency, or file sequence association between code blocks. The function call relationship is represented by adding weighted edges between nodes.

5. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The metadata layer stores and records attributes such as the location range of the code block in the source file, the physical path of the package to which it belongs, and visibility on the DAG node or edge.

6. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The method supports metadata tag-based search queries, and can filter and retrieve qualified code blocks by specifying tag values ​​(e.g., "AST type = loop structure"). It also allows the target code block to be retrieved based on the combination of block tags and associated attributes, and returns the corresponding block node identifier.

7. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The hot and cold data management strategy adopts a multi-level storage concept. Frequently accessed hot data is stored in high-performance components, and fine-grained blocks of hot data are maintained to support incremental updates later. Infrequently accessed cold data is merged according to the aggregation strategy and stored in low-cost storage.

8. The distributed storage method based on source code semantic segmentation according to claim 1, characterized in that: The DAG module is constructed and stored at the node end in a topologically sorted or BDS algorithm-optimized order to store semantic blocks.

Images