Massive open source code storage method and system based on atomization and bidirectional index

Abstract

The application discloses a kind of mass open source code storage method and system based on atomization and bidirectional index, its steps include: for the software code data of access, it is split into unique, immutable code atom is stored to the source code compression storage structure based on content addressing and constructs index;The source code compression storage structure includes three regions: metadata area, atomic code pool and multidimensional index area;The metadata area is used for format version number, storage statistical information, the physical offset address and length of atomic code pool, and the physical offset address and length of multidimensional index area;The atomic code pool is used to store the code atom;The multidimensional index area is used to store version reconstruction forward index and code fragment reverse index;The statistical information includes software code item number, software code version number and code atom total number.The application has remarkable advantages and beneficial effects in storage efficiency, access performance, analysis ability and system architecture etc.

Description

Technical Field

[0001] This invention belongs to the field of computer data storage and management technology, and relates to massive data storage, data compression, and data indexing technologies. Specifically, it relates to a method and system for storing massive open-source code based on atomicity and bidirectional indexing. Background Technology

[0002] In recent years, open-source software has become the foundation of modern information technology. To ensure the security, compliance, and stability of the software supply chain, large-scale open-source software supply chain infrastructure platforms have begun to be built. Such platforms need to crawl, aggregate, store, and analyze massive amounts of open-source software data from code hosting platforms (such as GitHub) worldwide. Among these data, the source code of open-source software (including its complete iteration history) occupies the largest portion of storage space.

[0003] Current mainstream open-source software code storage solutions primarily rely on general-purpose file systems or object storage (such as S3, HDFS) to store large amounts of compressed code packages (such as Gzip), or directly store raw repository data from code version control systems (such as Git). These solutions mainly focus on data "backup" and "archiving," with relatively weak data management and analysis capabilities, making them ill-suited for the application demands of modern open-source software supply chain infrastructure platforms that require in-depth analysis, security auditing, and compliance checks of massive amounts of code. Developing a new storage management method suitable for massive, multi-version, and highly redundant open-source software code is a prerequisite for achieving online analysis and supply chain security governance.

[0004] In terms of massive code version management, the work most relevant to this invention is Git's object model and its Packfile storage mechanism. As a distributed version control system, Git implements a content-addressable file storage and retrieval system at its core. It stores file content as blob objects, uniquely identified by their SHA-1 hash values. To save space, Git packages multiple objects into a Packfile (.pack file) and uses incremental (i.e., difference) compression during the packaging process. That is, a file version can be stored as the increment (i.e., difference) between it and another "base object".

[0005] First, the main drawback of existing mainstream general-purpose file systems or compressed package storage solutions is that they are completely "code-agnostic." They cannot understand the version iteration relationships and content duplication of the code, and cannot perform any form of content retrieval and analysis, making them unsuitable for the application scenarios of this invention.

[0006] Secondly, regarding the Git Packfile mechanism, which is most relevant to this invention, although it achieves efficient version storage within a single project, when applied to an open-source software supply chain infrastructure platform that aggregates millions or even more open-source projects worldwide, it has the following significant shortcomings and serious defects in data management.

[0007] (1) Existing work (Git) only solves the redundancy problem "within a project", but does not solve the global redundancy problem "across projects". Git's compression optimization is within a project (i.e., Intra-Project). If two different projects A and B both contain the same file in their code base, the two projects will generate their own Packfiles when packaging, resulting in the same file being physically stored twice or more. For a global platform that aggregates millions or even more projects, this cross-project (i.e., Inter-Project) redundant storage will cause huge waste of space.

[0008] (2) Existing work (Git) sacrifices access efficiency for compression, resulting in "incremental chain" access latency. Git's incremental compression mechanism leads to low access efficiency. To obtain the content of a file from a specific historical version, if the file is stored as an increment, the system must first load the base object and then apply a long "incremental chain" to finally reconstruct the target file. When version iterations are rapid and the increment chain is long, this reconstructing process consumes a lot of CPU and I / O resources, resulting in high access latency.

[0009] (3) Existing Git implementations lack content analysis capabilities and cannot perform reverse indexing of code snippets. The most critical flaw is that Git's model is a forward index (i.e., finding "file content" from "projects and versions"), completely lacking a reverse index to trace all occurrences of a "file content" or "code snippet." When the platform needs to perform security analysis (e.g., finding which projects and versions a known vulnerable code snippet appears in) or compliance analysis (e.g., tracing a specific license clause snippet), existing technology can only perform a full scan of all projects, all versions, and all files. This is completely infeasible with massive amounts of data, resulting in virtually zero timeliness (immediacy) of the analysis. Summary of the Invention

[0010] To address the problems existing in the prior art, the purpose of this invention is to provide a method and system for storing massive open-source code based on atomicity and bidirectional indexing. It is a data storage structure, storage writing and access method and system for large-scale, multi-version, and multi-source aggregated open-source software source code.

[0011] Open source software has become the cornerstone of the global digital economy, but its supply chain faces severe challenges in terms of security, compliance, and stability. To address these challenges, building an open source software supply chain infrastructure platform capable of aggregating, storing, and analyzing massive amounts of global open source software code has become an important development direction. In such platforms, the system not only needs to back up and archive massive amounts of code (PB-level or larger), but also needs to perform in-depth, real-time analysis, such as global vulnerability scanning, license compliance checks, and Software Component Analysis (SBOM).

[0012] Currently, there is a lack of systematic work on the storage and analysis of massive amounts of open-source software code, making it difficult to meet the real-time, intelligent, and global requirements of open-source software supply chain governance. This invention addresses these core needs by proposing and designing an efficient compression mechanism and storage method optimized for open-source software code storage. Through an innovative architecture that separates indexes and data, it supports high-level compression and efficient analysis of global code, ultimately achieving in-depth governance of open-source software code repositories and providing fundamental support for supply chain security.

[0013] 1. This invention innovatively proposes an archive data storage structure specifically for open-source software code. This format logically divides files into three distinct areas: the "Metadata Area," the "Atom Pool," and the "Multi-Dimensional Index Area." The "Atom Pool," as a purely data area, is responsible for physical storage; the "Multi-Dimensional Index Area," as a purely index area, is responsible for organizing the logical view.

[0014] The problem solved: It overcomes the fundamental flaws of existing technologies (such as Git Packfile) that mix index information (e.g., tree objects, commit objects) with data (blob objects), and prevent data sharing between projects. This separated architecture is a prerequisite for achieving global deduplication, independent expansion of indexes and data, and efficient access.

[0015] 2. This invention proposes a content-addressable global "atomic code pool" data layout. This layout is a content-addressable, append-only key-value store. The key is a globally unique hash value (AtomID) of the code atomic content (AtomData), and the value is CompressedAtomData after two levels of compression (such as Zstd).

[0016] Problem Solved: This is the core of achieving the global, cross-project deduplication function of this invention. Any code snippet, regardless of its source project, version, or file path, will be physically stored only once in the "atomic code pool" as long as its content is the same (same AtomID). This directly solves the serious cross-project redundancy problem caused by project isolation in the background technology.

[0017] 3. This invention proposes a multi-granularity code atomization method.

[0018] Deterministic file atomic mining: Utilizing file content data (blob objects) in version control systems (such as Git) to achieve file-level, deterministic multi-version duplicate mining.

[0019] Data-driven fragment atom mining: For text code files, use abstract syntax tree-based parsing to segment them according to semantic boundaries such as functions and methods; or use algorithms such as Content-Defined Chunking (CDC) to split the file into finer-grained "fragment atoms".

[0020] Problem Solved: This method balances efficiency and granularity. File atomic mining can efficiently process files that have not changed between versions; fragment atomic mining can delve into the internal structure of files, capturing fragment-level repetitions caused by "copy-paste" or code library reuse, achieving more significant compression, and laying the foundation for fragment-level analysis.

[0021] 4. This invention constructs two independent indexes in the "multidimensional index area" for efficient access and analysis of bidirectional indexes.

[0022] Version Refactoring Forward Index (Index A): A mapping from Key: (ProjectID, VersionID, FilePath) to Value: AtomManifest. AtomManifest is an ordered list of Fragment-AtomIDs that records the "recipe" required to refactor the file.

[0023] Inverted index for code snippet (Index B): An inverted index from Key: Fragment-AtomID to Value: PostingList (a list of locations). PostingList records all Locations (ProjectID, VersionID, FilePath, Offset) where the code snippet has appeared.

[0024] Problems Solved: Forward indexing completely solves the access latency problem caused by the "incremental chain" in the background technology, making the reconstruction of any historical version of the file a single index lookup and several parallel atomic data block reads, achieving efficient, low-latency on-demand access. Reverse indexing is a major innovation of this invention, solving the fatal flaw of the complete lack of content reverse lookup capability in the background technology. It transforms the analysis operation of full-database code scanning (such as vulnerability finding) into a near O(1) index query, greatly improving analysis efficiency.

[0025] 5. This invention proposes an artificial intelligence-based semantic association index as an enhancement to the inverted index. This invention utilizes code embedding models (such as CodeBERT) to convert code snippets into high-dimensional vectors and constructs vector indexes (such as FAISS and HNSW).

[0026] The problem solved: It has made a leap from "exact text matching" to "semantic similarity matching", and can discover "variants" of code (such as vulnerable code that only modifies variable names or comments), which greatly enhances the depth and breadth of security analysis and code clone detection.

[0027] The technical solution of this invention is a method for storing massive open-source code based on atomicity and bidirectional indexing. The steps include: for the accessed software code data, splitting it into unique, immutable code atoms and storing them in a content-addressable source code compression storage structure and building an index; the source code compression storage structure includes three areas: a metadata area, an atomic code pool, and a multidimensional index area; wherein, the metadata area is used for format version numbers, storage statistics, the physical offset address and length of the atomic code pool, and the physical offset address and length of the multidimensional index area; the atomic code pool is used to store the code atoms; the multidimensional index area is used to store version reconstruction forward indexes and code fragment reverse indexes; the statistics include the number of software code projects, the number of software code versions, and the total number of code atoms.

[0028] A preferred method for splitting software code data into unique, immutable code atoms is as follows: For open-source software code projects managed by a version control system, traverse the complete commit history of the open-source software code project, access each commit object and its associated tree object, and finally obtain all blob objects under the commit version snapshot; calculate the hash value of each blob object as the globally unique identifier AtomID of the blob object; if the blob object is a text-type code file, use a code parser to parse the blob object into an AST, and use the definition boundaries of "function", "method" or "class" as the splitting points to split the blob object into multiple semantically relatively complete code fragments; if the code parser cannot parse the blob object, use line-by-line or content-based block segmentation to split the blob object into multiple fragments as code fragments; then calculate the hash value of each code fragment as the unique identifier Fragment-AtomID_i of the fragment.

[0029] Preferably, the method for storing code atoms in the atomic code pool is as follows: First, query whether the globally unique identifier AtomID of the current code atom already exists in the atomic code pool. If it does not exist, compress the current code atom to obtain compressed data CompressedAtomData, then append the key-value pair [AtomID, CompressedAtomData] to the end of the atomic code pool and update the metadata; if the globally unique identifier AtomID of the current code atom already exists in the atomic code pool, then abandon the storage operation for the current code atom.

[0030] Preferably, the method for constructing the version refactoring forward index is as follows: A key-value storage structure is constructed, wherein a key is generated based on the unique project identifier ProjectID, version identifier VersionID, and file path of the blob object, and the value is based on the code atommanifest of the blob object; the code atommanifest is an ordered list that records the globally unique identifiers AtomID of all code atoms required to refactor the blob object; the key and value corresponding to each blob object are written into the key-value storage structure to obtain the version refactoring forward index.

[0031] Preferably, the method for constructing the reverse index of the code fragment is as follows: construct an inverted index, using the unique identifier Fragment-AtomID of the code fragment as the key of the code fragment, and the PostingList of the occurrence positions of the code fragment as the value of the code fragment; write the key of each code fragment and its corresponding value into the inverted index to obtain the reverse index of the code fragment.

[0032] Preferably, the location information of each occurrence of the code snippet is recorded using a structure Location (ProjectID, VersionID, FilePath, FragmentIndex / Offset); where ProjectID is the unique identifier of the project to which the code snippet belongs, VersionID is the version identifier of the code snippet, and FilePathFragmentIndex / Offset indicates which segment of the blob object the code snippet is or its starting offset.

[0033] Preferably, it also includes an artificial intelligence-based semantic association index as a supplement or enhancement to the reverse index of the code fragments; the method for constructing the semantic association index is as follows: first, each code fragment F_i is converted into a high-dimensional feature vector Vector_i; then, a vector index is constructed to store the mapping from the high-dimensional feature vector Vector_i to its corresponding structure Location or unique identifier Fragment-AtomID.

[0034] A method for querying massive amounts of open-source code, comprising the following steps: 1) When a user's query request to obtain the target file is received, the version reconstruction forward index is queried to obtain the key corresponding to the target file; 2) Obtain the corresponding value (Value) based on the key of the target file, which is the AtomManifest of the target file; 3) Based on the AtomManifest of the target file, query the atomic code pool in batches, obtain the corresponding code atoms, and concatenate them sequentially to form the target file and return it to the user.

[0035] A method for querying massive amounts of open-source code, comprising the following steps: 1) When a query request for the vulnerable code snippet Vulnerable_Snippet is received, calculate the unique identifier Fragment-AtomID_Vuln for that vulnerable code snippet Vulnerable_Snippet; 2) Query the inverted index of the code snippet to obtain the value corresponding to the key Key of the unique identifier Fragment-AtomID_Vuln, which is the PostingList of the occurrence positions of the vulnerable code snippet Vulnerable_Snippet; 3) Output an affected range report based on the PostingList, which lists the locations where the Vulnerable_Snippet code snippet appears.

[0036] A massive open-source code storage system based on atomicity and bidirectional indexing is characterized by comprising a code atomic processor, an index builder, a source code compressed storage structure, a code snippet real-time analyzer, and a data reconstructor. The code atom processor is used to split software code data into unique, immutable code atoms; The index builder is used to construct a version refactoring forward index and a code snippet reverse index based on the code atoms; The source code compressed storage structure includes three areas: a metadata area, an atomic code pool, and a multidimensional index area. The metadata area stores statistical information, the physical offset address and length of the atomic code pool, and the physical offset address and length of the multidimensional index area. The atomic code pool stores the code atoms. The multidimensional index area is used for version refactoring forward indexing and code fragment reverse indexing. The data reconstructor is used to, when receiving a user's query request to obtain a target file, query the version reconstruction forward index to obtain the key corresponding to the target file; obtain the corresponding value based on the key corresponding to the target file, which is the code atomic list AtomManifest of the target file; and query the atomic code pool in batches based on the code atomic list AtomManifest of the target file to obtain the corresponding code atoms and concatenate them sequentially to form the target file. The real-time code snippet analyzer, upon receiving a query request for a vulnerable code snippet (Vulnerable_Snippet), calculates the unique identifier (Fragment-AtomID_Vuln) of the vulnerable code snippet; queries the inverted index of the code snippet to obtain the value corresponding to the key (Key) of the unique identifier (Fragment-AtomID_Vuln), which is the PostingList of the locations where the vulnerable code snippet (Vulnerable_Snippet) appears; and outputs an affected scope report based on the PostingList of the locations where the vulnerable code snippet (Vulnerable_Snippet) appears.

[0037] The advantages of this invention are as follows: The efficient compression mechanism and storage method for optimizing open-source software code storage proposed in this invention, namely Content-Addressable Source Code Compression and Storage (CASCS), has significant advantages and beneficial effects in terms of storage efficiency, access performance, analysis capabilities, and system architecture compared with the background technology (i.e., storage mechanism of incremental compression within the project, represented by Git Packfile).

[0038] 1. Significantly improves storage compression ratio and achieves global data deduplication: Technical means: This invention adopts the data layout of "Atom Pool" and combines multi-granularity code atomization mining methods (including file level and fragment level).

[0039] Principle and Theoretical Effect: Background technology (Git Packfile) deduplication is project-wide. If a LICENSE file appears in 10,000 projects, it will be physically stored 10,000 times.

[0040] The "atomic code pool" of this invention is global and content-addressable. The aforementioned LICENSE files (or any code snippets) will be physically stored only once in the "atomic code pool" because they share the same AtomID.

[0041] In theory, for platforms that aggregate millions of open-source projects (in which there is a great deal of code reuse, vendor code copying, and file copying), the storage efficiency of this invention can be improved by several times, tens of times, or even more than that of the prior art.

[0042] Beneficial effects: Significantly reduces the storage costs of building large-scale open-source software code infrastructure. It compresses massive, redundant raw codebases into a concise, non-redundant global set of code atoms.

[0043] 2. Significantly reduce data access latency, enabling efficient on-demand refactoring: Technical means: The present invention designs a "version reconstruction forward index (Index A)", which stores a direct mapping from logical files to physical atom lists (AtomManifest).

[0044] Principles and theoretical effects: Background technology relies on "incremental chains" to reconstruct files. Accessing a file from a deep historical version may require CPU-intensive iterative calculations of dozens, hundreds, or even more increments, resulting in high I / O and CPU overhead, high latency, and unpredictability.

[0045] The access process of this invention is: one forward index lookup (to obtain the AtomManifest) + N parallel physical data block reads (N being the number of fragments). This process does not involve iterative calculations, resulting in low latency and stable, controllable performance.

[0046] Beneficial effects: It completely solves the high latency problem when accessing historical version files in the background technology. It provides high-performance, low-latency code access services for upper-layer applications (such as web code browsers, IDE plugins, etc.), greatly improving the user experience.

[0047] 3. Achieved content analysis and reverse lookup capabilities from "zero" to "instantaneous": Technical means: This invention has created a unique "code snippet reverse index (Index B)" and an optional "semantic association index".

[0048] Principles and theoretical effects: The background technology completely lacks the ability to trace the location from the content. Performing a "full-database vulnerability signature scan" requires traversing all Packfiles on the platform and decompressing all blobs, which is an O(N) (N is the total amount of code) analysis task that is almost impossible to complete.

[0049] This invention transforms this analysis operation into a single O(1) or O(logK) inverse index lookup (where K is the total number of atoms). Analysts only need to calculate the AtomID of the target code snippet to instantly (typically in milliseconds) obtain a list of all locations where it appears in the global codebase.

[0050] Beneficial effects: This is the most revolutionary advantage of this invention. It provides a powerful technical foundation for key functions such as security analysis (vulnerability situation awareness, poisoning risk warning), compliance auditing (license fragment tracking), and software composition analysis (SBOM) of the open-source software supply chain, making "real-time analysis of the entire library" possible, which was previously impossible.

[0051] 4. It provides a highly scalable and maintainable system architecture: Technical means: The present invention adopts an architecture that completely separates the index from the data ("atomic code pool" and "multidimensional index area" are separated).

[0052] Principles and theoretical effects: The background technology's packagefile is tightly coupled and project-based, making it difficult to perform global optimization and independent expansion.

[0053] In the architecture of this invention, the "atomic code pool" is append-only and immutable, making it ideal for deployment on high-throughput, highly scalable distributed object storage systems (such as S3). The "multidimensional index area" is mutable and access-intensive, suitable for deployment on high-performance indexing services or databases (such as databases or storage systems based on LSM-Tree or B-Tree).

[0054] Beneficial effects: It achieves separation of storage and computation, and separation of read and write operations. The system's storage layer and index layer can be scaled out independently according to their respective load characteristics, greatly improving the scalability, maintainability, and robustness of the entire infrastructure platform. Attached Figure Description

[0055] Figure 1 This diagram illustrates the process of accessing, storing, and managing open-source software code data. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0057] In open-source software supply chain infrastructure platforms, systems need to aggregate multi-source, multi-version code repositories from sources such as GitHub. This code data is characterized by massive volume, high versioning, and high redundancy. In this application scenario, traditional general-purpose storage (such as Gzip archives) or version control systems (such as Git) cannot effectively meet the requirements of functionality (inability to analyze) or performance (storage redundancy, slow access). Instead, a new storage management model that separates indexing and data, specifically designed for the characteristics of code, is required.

[0058] In the system proposed in this invention, code data undergoes continuous abstraction processing along the path of "code repository -> code atom -> index entry" before it is imported or used. The Content-Addressable Source-code Compressed Storage (CASCS) structure designed in this invention stores data at different levels of abstraction, with each area having its own responsibilities, collectively completing the storage of the global code repository.

[0059] A diagram illustrating open-source software code data access, storage, and processing is shown below. Figure 1As shown, data from multiple code data sources / code hosting platforms (such as GitHub and Gitee) undergo "code atomization" and "index building" operations when accessed into the massive software code storage system of this invention, ultimately generating CASCS storage structure data which is stored in the underlying system. When a data access and analysis request is received, this CASCS structure data undergoes data reconstruction and / or real-time code fragment analysis, and the returned result data provides services to the upper layers (such as security analysis and code retrieval).

[0060] The Atom Pool is a physical storage area. It stores all the unique, immutable "code atoms" that have been split.

[0061] The multidimensional index area is a logical view area. It stores the indexes that show how to reconstruct the logical file from the code atoms and how to look up the logical location from the code atoms.

[0062] The metadata area is responsible for storing statistical information and indexing the two areas mentioned above.

[0063] The present invention can be implemented as a system, which includes a data writing (construction) method and a data reading (access) method.

[0064] 1. Overall System Architecture This invention defines a content-addressable source code compression storage structure (CASCS), which is logically divided into three regions: A. Metadata Area: Stores the Magic Number (used to identify the CASCS format), the format version number, and global statistics (such as the total number of atoms, the total number of items, and the total number of versions). Stores the physical offset address and length of the "atomic code pool" and the "multidimensional index area"; Storage Bloom filters are used to quickly determine whether a given AtomID already exists in the "atomic code pool" when data is written, thereby reducing unnecessary index queries.

[0065] B. Atom Pool - (Data Layout Area) This area is the core of the data layout of this invention and is the physical storage area for all unique code content.

[0066] This region is content-addressable, and its data layout is a collection of key-value pairs: [AtomID_1, CompressedAtomData_1], [AtomID_2, CompressedAtomData_2], ... AtomID: is the hash value (preferably SHA-256) of AtomData (the original code snippet content), serving as its globally unique identifier.

[0067] CompressedAtomData is binary data obtained by compressing the original AtomData using standard, efficient compression algorithms (such as Zstd, Snappy, LZ4).

[0068] This area is written in an append-only manner, conforming to the immutable nature of code version data. Any code snippet (regardless of its granularity, whether it is a file or a function) will only be physically stored once in this area, provided that its AtomID is the same.

[0069] C. Multi-Dimensional Index Area This area is responsible for storing all index structures, which are used to connect logical views and physical data.

[0070] It mainly consists of the "Version Refactoring Forward Index" (Index A) and the "Code Snippet Reverse Index" (Index B), which are detailed below.

[0071] 2. System Construction and Data Writing Methods When a new open-source project (or batch of projects) (such as a complete Git repository) needs to be added to the system, the system performs the following steps: Step 1: Code Atomization and Repetition Detection This step is fundamental to deduplication and indexing, employing a combination of deterministic and data-driven approaches.

[0072] 1.1 Deterministic File Atom Mining: The system traverses the complete commit history of this open-source project's codebase managed by a version control system (such as Git), accesses each version commit object and its associated directory tree object, and finally obtains all source code file data objects (blob objects, i.e., the content of specific software project code files) under the snapshot corresponding to that version commit.

[0073] The system calculates the SHA-256 hash value of the file data object (blob) content, and records it as File-AtomID.

[0074] This method leverages the features of version control systems (such as Git) to deterministically identify all files that have not changed between versions and branches, enabling efficient multi-version duplicate code mining.

[0075] 1.2 Data-Driven Fragment-Atom Mining: For text-based code files (such as .c, .java, .py, etc.), the system (especially for new or changed blobs) further breaks them down into smaller-grained "code atoms" to capture code reuse within and between files.

[0076] Segmentation method: First, based on Abstract Syntax Tree (AST) segmentation: The system uses a code parser (such as ANTLR, Tree-sitter) to parse the code file into an AST, and uses the definition boundaries of "function", "method" or "class" as the segmentation points to divide the file into multiple semantically relatively complete segments F_1, F_2, ... F_n as code fragments.

[0077] Secondly, text-based segmentation: For unparseable text files (such as Makefile, .conf) or as a degradation strategy, segmentation can be performed by line or content definition (such as Rabin-Karp rolling hash). The resulting segments are also used as code fragments F_i in subsequent processing.

[0078] The system calculates the SHA-256 hash value of each code fragment F_i, denoted as Fragment-AtomID_i.

[0079] Step 2: Populate the atomic code pool In step 1, for each generated AtomID (whether File-AtomID or Fragment-AtomID) and its corresponding raw data AtomData: The system first queries (using a Bloom filter in the metadata area) to determine whether the AtomID already exists in the "atomic code pool": If it does not exist: The system first performs a compression algorithm (such as Zstd) on AtomData to obtain CompressedAtomData, then appends the key-value pair [AtomID, CompressedAtomData] to the end of the "atomic code pool" and updates the metadata.

[0080] If it already exists: no physical storage operation will be performed.

[0081] Beneficial effects: This step ensures that any duplicate code content, whether across projects, versions, or files, is physically stored only once globally, achieving significant storage compression.

[0082] Step 3: Version Reconstruction of Forward Index (Index A) Construction This index is used to solve the "access latency problem" and enable fast reconstruction of any file.

[0083] Index structure: A key-value storage structure (e.g., implemented using a B-Tree or LSM-Tree).

[0084] Key: (ProjectID, VersionID, FilePath) ProjectID is the unique identifier for the project, VersionID is the version identifier (such as Commit Hash), and FilePath is the file path.

[0085] Value: AtomManifest (Atom List) AtomManifest is an ordered list that records the IDs of all atoms required to reconstruct the file: [Fragment-AtomID_1, Fragment-AtomID_2, ... Fragment-AtomID_n].

[0086] Build process: In step 1.2, when the file is split into F_1...F_n, the system uses (ProjectID, VersionID, FilePath) as the key and this ordered list of IDs [Fragment-AtomID_1, ...,Fragment-AtomID_n] as the value, and writes it into index A.

[0087] (Note: For an unsplit binary file, its AtomManifest can be a single File-AtomID.)

[0088] Step 4: Building the reverse index (Index B) for the code snippet This index is used to solve the "analysis efficiency problem" and enable real-time reverse lookup of code snippets.

[0089] Index structure: An inverted index.

[0090] Key: Fragment-AtomID Value: PostingList (a list of locations where the location appears), i.e., [Location_1, Location_2, ... The Location field is a structure that precisely records the location where the fragment appears, for example: (ProjectID, VersionID, FilePath, FragmentIndex / Offset). FragmentIndex / Offset specifies which fragment in the file the fragment belongs to or its starting offset.

[0091] Build process: In step 3, when the system records AtomManifest, it will obtain the corresponding PostingList for each Fragment-AtomID_k in the manifest and append the new location information (ProjectID, VersionID, FilePath, FragmentIndex / Offset) to the list.

[0092] 3. Examples of Semantic Association Based on Artificial Intelligence Based on the above technical solution, the present invention may also include an artificial intelligence-based semantic association index as a supplement or enhancement to the "code snippet inverted index (Index B)".

[0093] Construction method: Step 1. Vector Generation: In "Step 1: Code Atomization", while calculating the hash Fragment-AtomID for code fragments F_i (especially semantic units such as functions and methods), the system also uses a pre-trained code embedding model (such as CodeBERT, GraphCodeBERT, etc.) to convert it into a high-dimensional feature vector Vector_i.

[0094] Step 2. Vector Index Construction: The system constructs a dedicated vector index (such as FAISS, HNSW, ANNOY, etc.), which stores the mapping from Vector_i to its Location (or Fragment-AtomID).

[0095] How to use: Step 1. When analysts need to find a “variant” of a vulnerability (i.e., the same in meaning but different in variable name, comments, or format), they can provide a sample fragment; Step 2. The system converts the sample fragment into a query vector Vector_q; Step 3. By querying the vector index, the system can recall the Top-K code snippets in the vector space that are closest to Vector_q and their positions.

[0096] Beneficial effects: This embodiment achieves a leap from "exact text matching" to "semantic similarity matching", greatly enhancing the ability of security analysis and code clone detection.

[0097] 4. System Access and Analysis Methods The data access method of the present invention utilizes the above-described data layout and index structure: Scenario 1: Accessing specific files on demand (forward access, addressing the drawback (2)) When a user or an upper-layer application requests a specific file, such as (Project_A, Commit_Hash_X, 'src / main.c'): Step 1. The system queries the "forward index (Index A)" using the composite key Key: (Project_A,Commit_Hash_X, 'src / main.c'); Step 2. The system immediately obtains its Value, which is a list of AtomManifest, such as [Atom_H1, Atom_H2, Atom_H3]; Step 3. The system uses this ID list to perform batch queries on the "Atom Pool". Since AtomID is the key, the query efficiency is high (for example, assuming the Atom Pool is based on Hash or B-Tree storage). The system retrieves the CompressedAtomData corresponding to the three keys Atom_H1, Atom_H2, and Atom_H3; Step 4. The system decompresses these three data blocks in parallel, and then concatenates them (as in memory) strictly in the order of [H1, H2, H3] to form the complete contents of the 'src.main.c' file, and returns it to the user.

[0098] Beneficial effects: This process involves only one index lookup and N (N is the number of fragments) efficient data block reads, completely avoiding the iterative reconstruction of the "incremental chain" and resulting in extremely low access latency.

[0099] Scenario 2: Analyzing code snippets (reverse access, addressing shortcomings (3)) When the security analysis module needs to find a known vulnerable code snippet (e.g., a dangerous function implementation): Step 1. The system first performs the same "fragment atomization" algorithm (such as AST segmentation) as in step 1.2 on the Vulnerable_Snippet to calculate its Fragment-AtomID_Vuln; Step 2. The system queries the "inverted index (Index B)" using the Key: Fragment-AtomID_Vuln; Step 3. The system obtains its Value, i.e., PostingList, in real time (typically within milliseconds); this list contains a complete list of locations where this vulnerable fragment has appeared in all projects and versions globally: [(Proj_A, Commit_X, ...), (Proj_B, Commit_Y, ...), (Proj_C, Commit_Z, ...)]; Step 4. The analysis platform outputs an affected scope report based on this list without performing any time-consuming code scanning.

[0100] Beneficial effects: The full code scan, which originally took days or even months to complete with massive amounts of data, is transformed into a near O(1) index query, which greatly improves the timeliness of security analysis, supply chain "poisoning" risk perception, and compliance audit.

[0101] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for storing massive open source code based on atomization and bidirectional indexing, comprising the steps of: For the accessed software code data, it is split into unique, immutable code atoms, stored in a content-addressable source code compressed storage structure, and an index is built. The source code compression storage structure comprises three areas: a metadata area, an atomic code pool, and a multidimensional index area. The metadata area stores the format version number, statistical information, the physical offset address and length of the atomic code pool, and the physical offset address and length of the multidimensional index area. The atomic code pool stores the code atoms. The multidimensional index area stores the forward index for version refactoring and the reverse index for code fragments. The statistical information includes the number of software code projects, the number of software code versions, and the total number of code atoms. The method for constructing the version refactoring forward index is as follows: A key-value storage structure is constructed, where a key is generated based on the unique project identifier (ProjectID), version identifier (VersionID), and file path (FilePath) of the blob object; the value of the key is the code atom manifest (AtomManifest) of the blob object; the code atom manifest (AtomManifest) is an ordered list recording the globally unique identifier (AtomID) of all code atoms required to refactor the blob object; the key and value corresponding to each blob object are written into the key-value storage structure to obtain the version refactoring forward index. The method for constructing the reverse index of the code fragment is as follows: construct an inverted index, using the unique identifier Fragment-AtomID of the code fragment as the key of the code fragment, and the PostingList of the occurrence positions of the code fragment as the value of the code fragment; write the key of each code fragment and its corresponding value into the inverted index to obtain the reverse index of the code fragment.

2. The method of claim 1, wherein, The method for splitting software code data into unique, immutable code atoms is as follows: For open-source software code projects managed by a version control system, traverse the complete commit history of the open-source software code project, access each commit object and its associated tree object, and finally obtain all blob objects under the commit version snapshot; calculate the hash value of each blob object as the globally unique identifier AtomID of the blob object; if the blob object is a text-type code file, use a code parser to parse the blob object into an AST, and use the definition boundaries of "function", "method" or "class" as the split points to split the blob object into multiple semantically relatively complete code fragments; if the code parser cannot parse the blob object, use line-by-line or content-based block segmentation to split the blob object into multiple fragments as code fragments; then calculate the hash value of each code fragment as the unique identifier Fragment-AtomID_i of the fragment.

3. The method of claim 2, wherein, The method for storing code atoms into the atomic code pool is as follows: First, query whether the globally unique identifier AtomID of the current code atom already exists in the atomic code pool. If it does not exist, compress the current code atom to obtain compressed data CompressedAtomData, then append the key-value pair [AtomID, CompressedAtomData] to the end of the atomic code pool and update the metadata; if the globally unique identifier AtomID of the current code atom already exists in the atomic code pool, then abandon the storage operation for the current code atom.

4. The method of claim 1, wherein, The location information for each occurrence of the code snippet is recorded using a structure Location (ProjectID, VersionID, FilePath, FragmentIndex / Offset); where ProjectID is the unique identifier of the project to which the code snippet belongs, VersionID is the version identifier of the code snippet, and FilePath, FragmentIndex / Offset indicate which segment of the blob object the code snippet belongs to or its starting offset.

5. The method of claim 1, wherein, It also includes an AI-based semantic association index as a supplement or enhancement to the reverse index of the code fragments; the method for constructing the semantic association index is as follows: first, each code fragment F_i is converted into a high-dimensional feature vector Vector_i; then, a vector index is constructed to store the mapping from the high-dimensional feature vector Vector_i to its corresponding structure Location or unique identifier Fragment-AtomID.

6. A method for querying massive amounts of open-source code stored based on the method described in claim 1, comprising the following steps: 1) When a user's query request to obtain the target file is received, the version reconstruction forward index is queried to obtain the key corresponding to the target file; 2) Obtain the corresponding value (Value) based on the key of the target file, which is the AtomManifest of the target file; 3) Based on the AtomManifest of the target file, query the atomic code pool in batches, obtain the corresponding code atoms, and concatenate them sequentially to form the target file and return it to the user.

7. A method for querying massive amounts of open-source code stored using the method described in claim 1, comprising the following steps: 1) When a query request for the vulnerable code snippet Vulnerable_Snippet is received, calculate the unique identifier Fragment-AtomID_Vuln for that vulnerable code snippet Vulnerable_Snippet; 2) Query the inverted index of the code snippet to obtain the value corresponding to the key Key of the unique identifier Fragment-AtomID_Vuln, which is the PostingList of the occurrence positions of the vulnerable code snippet Vulnerable_Snippet; 3) Output an affected range report based on the PostingList, which lists the locations where the Vulnerable_Snippet code snippet appears.

8. A mass open source code storage system based on atomization and bidirectional indexing, characterized in that, This includes code atomic processors, index builders, source code compressed storage structures, code snippet instant analyzers, and data refactoring machines; The code atom processor is used to split software code data into unique, immutable code atoms; The index builder is used to construct a version refactoring forward index and a code snippet reverse index based on the code atoms. The method for constructing the version refactoring forward index is as follows: A key-value storage structure is constructed, where a key is generated based on the blob object's unique project identifier (ProjectID), version identifier (VersionID), and file path (FilePath), and the value is the blob object's code atom list (AtomManifest). The code atom list (AtomManifest) is an ordered list recording the globally unique identifiers (AtomIDs) of all code atoms required to refactor the blob object. The key and value corresponding to each blob object are written into the key-value storage structure to obtain the version refactoring forward index. The method for constructing the code snippet reverse index is as follows: An inverted index is constructed, where the unique identifier (Fragment-AtomID) of the code snippet is used as the key, and the posting list of the code snippet's occurrence positions is used as the value. The key and value of each code snippet are written into the inverted index to obtain the code snippet reverse index. The source code compressed storage structure includes three areas: a metadata area, an atomic code pool, and a multidimensional index area. The metadata area stores statistical information, the physical offset address and length of the atomic code pool, and the physical offset address and length of the multidimensional index area. The atomic code pool stores the code atoms. The multidimensional index area is used for version refactoring forward indexing and code fragment reverse indexing. The data reconstructor is used to, when receiving a user's query request to obtain a target file, query the version reconstruction forward index to obtain the key corresponding to the target file; obtain the corresponding value based on the key corresponding to the target file, which is the code atomic list AtomManifest of the target file; and query the atomic code pool in batches based on the code atomic list AtomManifest of the target file to obtain the corresponding code atoms and concatenate them sequentially to form the target file. The real-time code snippet analyzer, upon receiving a query request for a vulnerable code snippet (Vulnerable_Snippet), calculates the unique identifier (Fragment-AtomID_Vuln) of the vulnerable code snippet; queries the inverted index of the code snippet to obtain the value corresponding to the key (Key) of the unique identifier (Fragment-AtomID_Vuln), which is the PostingList of the locations where the vulnerable code snippet (Vulnerable_Snippet) appears; and outputs an affected scope report based on the PostingList of the locations where the vulnerable code snippet (Vulnerable_Snippet) appears.

Images