Multi-level intelligent storage of mass open source code and migration method between intelligent levels

Abstract

This invention discloses a method for migrating massive amounts of open-source code between multi-level intelligent storage and intelligent tiers. The steps include: constructing a logically unified but physically heterogeneous multi-level storage architecture to store data of varying popularity using storage media with different performance levels; collecting business flows from code hosting platforms on a code repository basis and generating time-series feature vectors for each repository; calculating the comprehensive activity score of the code repository based on the time-series feature vectors; generating migration instructions based on the comprehensive activity score and the storage tier where the code repository is located; performing the migration of the code repository according to the migration instructions and updating the code repository's metadata; and identifying hot files based on the code repository's historical access logs and loading them into the cache of the target storage tier to which the migration is being carried out. This invention effectively solves the problems of single evaluation and high migration costs in traditional solutions, achieving full-process automation and intelligence in storage operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of computer software and data storage technology, and relates to a method for multi-level intelligent storage of massive open-source code and migration between intelligent levels. Background Technology

[0002] Open-source code hosting platforms (such as GitHub and GitLab) store massive amounts of code data, and their project activity distribution exhibits a significant long-tail characteristic: highly active projects at the top (such as the Linux kernel) have frequent updates and numerous contributors, thus requiring high-performance storage to support real-time collaboration; while low-active projects at the bottom (such as experimental repositories that have been unmaintained for a long time) have not been updated for a long time, making them suitable for migration to low-cost archive storage. Some existing tiered storage solutions (such as patent CN116185284B) adopt a two-tier storage architecture based on data block activity. By monitoring the access frequency of data blocks, they automatically achieve dynamic data migration between the NVMe SSD high-level tier and the SATA SSD / HDD low-level tier without human intervention. However, this type of solution based on underlying data blocks and a single access metric still has the following shortcomings.

[0003] (1) Static storage tiers: Traditional hot and cold tiers usually rely on manually preset rules, which cannot detect dynamic changes in project activity (such as sudden peaks in community contributions) and make it difficult to adjust storage strategies in a timely manner.

[0004] (2) Single evaluation dimension: Existing activity evaluation models often only focus on a few indicators such as code submission frequency, thus failing to fully reflect the value of the project.

[0005] (3) High migration cost: In the existing solution, it is often necessary to migrate the entire dataset, which will lead to a serious I / O storm; and there is a lack of fine-grained migration units, which cannot optimize the migration of local areas such as subdirectories of the code repository, resulting in low efficiency during the migration process.

[0006] (4) The migration process affects business: Data migration often consumes a lot of network bandwidth and may reach its peak in a short period of time, thus affecting the access performance of normal business and lacking effective control over migration traffic.

[0007] (5) Lack of time decay and anomaly identification: The existing solution does not introduce a time decay factor, which causes historically active but recently stagnant projects to occupy high-performance storage resources for a long time; at the same time, it cannot identify the revival of "zombie" projects (for example, some projects have a surge in activity in a short period of time due to the patching of security vulnerabilities), making it difficult to adjust the storage level in a timely manner.

[0008] To address the above issues, it is necessary to provide a new technical solution to support more intelligent, granular, and efficient multi-level storage management. Summary of the Invention

[0009] To address the problems existing in the prior art, the purpose of this invention is to provide a method for migrating between multi-level intelligent storage and intelligent layers of massive open-source code.

[0010] This invention is based on semantic units such as code repositories and achieves dynamic storage management through multi-dimensional evaluation. Its core technical content is as follows: 1. Multimodal Dynamic Evaluation Mechanism: Innovatively integrates three dimensions: code activity, community collaboration, and code influence. It introduces a time decay factor to emphasize recent data and utilizes an LSTM neural network to automatically learn trends and dynamically adjust weights (e.g., automatically and significantly increasing influence weights when a security vulnerability is detected), achieving accurate and adaptive evaluation of project value.

[0011] 2. Hybrid triggering mechanism driven by both prediction and events: It includes regular migration based on scoring thresholds, and can also trigger migration instantly for sudden events (such as vulnerability announcements) or a surge in activity predicted by the model, without waiting for the regular cycle, thus enabling rapid response to sudden situations.

[0012] 3. Streaming Lossless Migration Solution: Utilizes Change Data Capture (CDC) technology to synchronize data in real time, ensures zero data loss during synchronization through dual write buffer queues, and achieves sub-second seamless switching using a distributed atomic switching protocol. Combined with log parsing and consistency verification, it guarantees business continuity during large-scale migrations.

[0013] 4. Fine-grained architecture and intelligent optimization: A four-tier storage architecture (cold, warm, hot, and extremely hot) is constructed, supporting fine-grained management at the repository or directory level. During migration, intelligent prefetching technology is used in conjunction with historical access patterns to preload hot files into the target layer cache, significantly reducing access latency.

[0014] The technical solution of this invention is as follows: A method for migrating between multi-level intelligent storage and intelligent hierarchies of massive open-source code, the steps of which include: Construct a logically unified but physically heterogeneous multi-level storage architecture to store data of varying popularity using storage media with different performance levels; The business flow of the code hosting platform is collected on a code repository basis, and a time-series feature vector of the code repository is generated. The overall activity score of the code repository is calculated based on the time series feature vector. Based on the overall activity score of the code repository and the storage level where the code repository is located, a migration instruction is generated; The migration instructions are used to migrate the code repository and update its metadata; and the historical access logs of the code repository are used to identify hot files and load them into the cache of the target storage level to which the migration is being carried out.

[0015] Preferably, the multi-level storage architecture includes an extremely hot layer for storing highly active code data, a hot layer for storing active projects, a warm layer for storing stable projects, and a cold layer for storing code identified as zombie code.

[0016] Preferably, the extreme heat layer uses an NVMe SSD storage cluster and is configured with DRAM memory or Redis as a first-level high-speed cache; the heat layer uses a distributed SSD storage pool and employs erasure coding strategy to segment and encode the data; the warm layer uses a high-density HDD disk array and enables LZ4 for real-time data compression; the cold layer uses an object storage system or Blu-ray storage repository and employs a high compression ratio algorithm to perform deep data compression.

[0017] Preferably, the method for generating the time series feature vector is as follows: Extract the timestamps of code commits, the change in lines of code per commit, and the dispersion of file changes within each time slice of the code repository as code activity dimension data; Extract the number of new issues, closed issues, comment interaction frequency, and average time from commit to merge for each time slice in the code repository as data for community collaboration. The data includes statistics on the growth rate of stars / forks of a code repository in each time slice, the number of downstream dependent projects, and whether the code repository is associated with unpatched high-risk security vulnerabilities, serving as data for influence and security dimensions. Normalize the code activity dimension data, community collaboration dimension data, and influence and security dimension data to generate feature vectors for the code repository in each time slice; arrange the feature vectors of the code repository in each time slice in order to generate the time series feature vector of the code repository.

[0018] Preferably, the method for calculating the overall activity score of the code repository is as follows: The code activity dimension data is processed using a time decay function to calculate the real-time popularity value of the code repository; Predict the predicted popularity value of the code repository based on its historical time series characteristics; Calculate the weighted value of security events in the code repository based on data from the impact and security dimensions; The overall activity score of the code repository is calculated by combining the real-time popularity value, the predicted popularity value, and the security event weighted value.

[0019] Preferably, the method for generating the migration instructions is as follows: If the overall activity score of the code repository is higher than the admission threshold of the storage level above the current storage level of the code repository, and the duration of the overall activity score exceeds the set anti-jitter window, then an upward migration instruction is generated; or if the code repository is marked as a sudden security event, then an upward migration instruction is generated. If the overall activity score of the code repository is lower than the retention threshold of the current storage level of the code repository, and it is predicted that the code repository will not have an upward trend in the near future, a downward migration instruction will be generated.

[0020] Preferably, before generating the migration instruction, the remaining capacity of the target storage level to which the migration is to be made is checked; if the capacity of the target storage level is insufficient, the code repositories with the lowest overall activity scores are forcibly migrated out in descending order of their overall activity scores, and the space is allocated to code repositories with higher overall activity scores.

[0021] A multi-level intelligent storage and intelligent hierarchy migration system for massive open-source code is characterized by including a storage resource pool construction module, a feature extraction module, a scoring module, a migration instruction generation module, and a migration execution module. The storage resource pool construction module is used to construct a logically unified but physically heterogeneous multi-level storage architecture, which uses storage media with different performance to store data with different popularity. The feature extraction module is used to collect the business flow of the code hosting platform on a code repository basis and generate the time series feature vector of the code repository. The scoring module is used to calculate the overall activity score of the code repository based on the time series feature vector; The migration instruction generation module is used to generate migration instructions based on the overall activity score of the code repository and the storage level where the code repository is located. The migration execution module is used to perform migration on the code repository according to the migration instructions and update the metadata of the code repository; and to identify hot files according to the historical access logs of the code repository and load them into the cache of the target storage level to which the migration is to take place.

[0022] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0023] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program implements the above-described method when executed by a processor.

[0024] The advantages of this invention are as follows: This invention effectively solves the problems of single evaluation and high migration costs in traditional solutions by using semantic-level layering and a closed-loop feedback mechanism. It not only significantly reduces storage costs through hot / cold tiering but also greatly improves the performance of high-frequency access items through intelligent caching, achieving full automation and intelligence in storage operation and maintenance. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention. Figure 2 This is a flowchart of a method according to an embodiment of the present invention. Figure 3 This is a system block diagram of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0027] An optional embodiment of the present invention provides a method for migrating between multi-level intelligent storage and intelligent hierarchies of massive open-source code, the steps of which include: Construct a logically unified but physically heterogeneous multi-level storage architecture to store data of varying popularity using storage media with different performance levels; The business flow of the code hosting platform is collected on a code repository basis, and a time-series feature vector of the code repository is generated. The overall activity score of the code repository is calculated based on the time series feature vector. Based on the overall activity score of the code repository and the storage level where the code repository is located, a migration instruction is generated; The migration instructions are used to migrate the code repository and update its metadata; and the historical access logs of the code repository are used to identify hot files and load them into the cache of the target storage level to which the migration is being carried out.

[0028] like Figure 2 As shown, one embodiment of the present invention uses a large hosting platform containing millions of code repositories as a scenario to introduce the method flow of the present invention, which includes the following steps: Step S1: Construct a four-level heterogeneous storage resource pool based on media characteristics First, based on the performance and cost differences of physical storage media, the system constructs a logically unified but physically heterogeneous four-tier storage architecture. This four-tier storage architecture forms the physical foundation for subsequent data migration. Tier-Extreme: As the highest performance tier, it employs an NVMe SSD storage cluster and is configured with DRAM memory or Redis as a primary high-speed cache. This tier is used to store ultra-high-activity code data accessed via the RDMA network, and the system is configured with 3 replicas for redundancy to ensure data reliability.

[0029] Tier-Hot: As a high-performance layer, it employs a distributed SSD storage pool (such as one built on Ceph). This layer stores projects that are normally active. It uses erasure coding (such as EC 3+1) strategies to segment and encode the stored data, thereby reducing redundancy while ensuring data fault tolerance, thus balancing performance and cost.

[0030] Tier-Warm: As a medium-performance tier, it employs a high-density HDD disk array with LZ4 real-time compression enabled in the file system. This tier is used to store stable projects that have not been updated recently but have a small number of reads.

[0031] Tier-Cold: This layer serves as an archiving layer, interfacing with object storage systems (such as S3 protocol storage) or Blu-ray repositories. It stores long-frozen "zombie" code, employing high-compression algorithms (such as Brotli) for deep compression.

[0032] Step S2: Collect multimodal features and generate time series feature vectors (input stage) The system monitors the business flow of the code hosting platform in real time, collecting raw data on a "code repository" basis. The collected data serves as input for subsequent evaluation models, specifically including three dimensions: Code activity dimension data: The system extracts the timestamp of code commits, the change in the number of lines of code in a single commit, and the dispersion (entropy value) of file changes by parsing the Git commit log.

[0033] Community collaboration data: The system monitors the Issue tracker and Pull Request (PR) system to extract the number of new issues created, closed, and comment interaction frequency in the code repository per unit time, as well as the average time taken from PR submission to merging.

[0034] Influence and security data: The system tracks the growth rate of stars / forks of the code repository, the number of downstream dependent projects, and connects to the CVE (Common Vulnerability Disclosure) database in real time to monitor whether the code repository is associated with unpatched high-risk security vulnerabilities.

[0035] The system normalizes the data across the aforementioned dimensions. Since the subsequent LSTM network requires time-series data as input, the system divides continuous time into multiple fixed-length time slices (e.g., 24 hours per slice). For each code repository, the system first extracts features from the data across the aforementioned dimensions collected within a single time slice, generating a feature vector for that code repository within that time slice. Subsequently, the system extracts feature vectors from the current and N consecutive historical time slices (e.g., the past 30 days), merges and arranges them chronologically to generate a complete time-series feature matrix for the code repository, which is then transmitted to the next step.

[0036] Step S3: LSTM-based activity score calculation and trend prediction (processing stage) The system receives the time-series feature vector output from step S2 and uses a pre-trained Long Short-Term Memory (LSTM) network model to calculate the "comprehensive activity score" of the code repository. The calculation process includes specific weighting and prediction logic: Time-decay weighted average: For "code activity dimension data" and "community collaboration dimension data," the system uses a time-decay function for processing. Specifically, behaviors closer to the current time have higher weights, thus calculating a "real-time heat value" reflecting the current intensity of development and collaboration. The specific calculation formula is as follows: in: This is the real-time popularity value; t is the sequence number of the latest time slice; t is the sequence number of the historical time slices (from 1 to 1). ); The time decay weighting factor, The preset decay constant makes older data ( The larger the (the greater) the weight, the more exponentially the decrease; Score the normalized "code activity dimension" for this code repository within time slice t; A normalized score for the "community collaboration dimension" within time slice t; and These are the weight coefficients for the corresponding dimensions (e.g.) , ).

[0037] Trend Prediction: Input the time-series feature vector of the current target code repository generated in step S2 into the pre-trained LSTM network. The LSTM network uses its memory units to analyze the changes in the data of this specific repository over time (e.g., identify patterns such as "a surge in submissions every Friday afternoon" or "a surge in access within 24 hours after a vulnerability is disclosed"), and outputs the "predicted popularity value" for the next time window (e.g., the next 24 hours).

[0038] Security event weighting: When the input data contains the "high-risk CVE vulnerability" tag, the model will automatically trigger the weight reset mechanism to forcibly increase the weight of the "impact and security dimension" (e.g., increase it to 50%) to ensure that security remediation-related code can get the highest score.

[0039] Ultimately, the model combines the "real-time popularity value," "predicted popularity value," and "security event weighted value" to output a comprehensive activity score between 0 and 100.

[0040] Step S4: Generate migration decision instructions (decision phase) The system compares the overall activity score output in step S3 with the storage level of the current code repository and generates migration instructions based on preset threshold rules: Jump rule determination: Upgrade migration: If the overall activity score is higher than the admission threshold of the target high level (e.g., score > 90 points, and currently located in the hot or warm layer), and the high score state lasts for more than the anti-jitter window (e.g., 10 minutes), or is marked as a "sudden security event" by the model, the system immediately generates an "upgrade migration" instruction.

[0041] Downgrade migration: If the overall activity score is lower than the retention threshold of the current storage level (e.g., score < 20 points), and the model predicts no upward trend in the next week (the specific judgment condition is: the "predicted popularity value" of each time slice output by the LSTM model in the next week is continuously less than the retention threshold, and the slope of the predicted popularity value changing with time is in the zero or negative range), the system generates a "downgrade migration" instruction.

[0042] Capacity constraint verification: Before generating instructions, the system checks the remaining capacity of the target storage layer. If the target storage layer (such as the hottest layer) has insufficient capacity, it sorts the code repositories by overall activity score in descending order and forcibly moves out the code repositories with the lowest overall activity scores in the target storage layer, freeing up space for code repositories with higher overall activity scores, ensuring that high-value data has priority access to high-performance resources.

[0043] Step S5: Perform CDC-based streaming lossless migration (execution phase) In response to the migration command generated in step S4, the migration execution module initiates a streaming data transfer process to ensure uninterrupted business operations and no data loss during the migration. Full snapshot copy: Establish a channel between the source storage level and the target storage level, and use file system snapshot technology to copy the base data files of the code repository to the target level.

[0044] Incremental Data Capture (CDC): During full replication, the system enables change data capture. This feature monitors and records all new write operations to the code repository (such as new code pushed by git push) in real time, writing these incremental operations to a temporary in-memory buffer queue.

[0045] Incremental catch-up and dual write: After full copying is complete, the system reads incremental operations from the buffer queue and replays them at the target storage level. When the backlog of data in the queue approaches zero, the system enters "dual write mode," where newly arriving write requests are simultaneously written to both the source and target storage levels to ensure complete data consistency at both ends.

[0046] Atomic Switching: Utilizing a distributed metadata service (such as one based on Etcd), an atomic pointer switch operation is performed. The system updates the metadata storage location field of the code repository to the target storage tier address and increments the version number. This operation is completed in milliseconds, and subsequent client requests will be automatically routed to the target storage tier.

[0047] Intelligent prefetching and cleanup: Intelligent prefetching: After the migration is complete, the system analyzes the historical access logs of the code repository, identifies hot files (such as README.md and HEAD pointer files), and immediately loads them into the Redis cache of the target storage layer to eliminate the first access delay after the migration.

[0048] Delayed reclamation: Data in the source storage layer is retained for 24 hours as a backup (tombstone mechanism). If there are no abnormal errors, it is asynchronously deleted after 24 hours to free up space.

[0049] Example of effect description: Through the above steps, this invention enables precise dynamic layering. For example, when a security vulnerability is discovered in the Linux kernel repository (signal acquired in step S2), the LSTM model predicts a large number of patch submissions and code pulls in the future (score spikes in step S3). Within minutes, the system migrates the vulnerability from the HDD warm layer to the NVMe hot layer (steps S4 and S5), reducing IO latency from 50ms to 0.5ms when global developers collaborate on vulnerability fixes, thus avoiding storage performance bottlenecks.

[0050] like Figure 3As shown, an optional embodiment of the present invention provides a multi-level intelligent storage and intelligent hierarchy migration system for massive open-source code, characterized in that it includes a storage resource pool construction module, a feature extraction module, a scoring module, a migration instruction generation module, and a migration execution module; The storage resource pool construction module is used to construct a logically unified but physically heterogeneous multi-level storage architecture, which uses storage media with different performance to store data with different popularity. The feature extraction module is used to collect the business flow of the code hosting platform on a code repository basis and generate the time series feature vector of the code repository. The scoring module is used to calculate the overall activity score of the code repository based on the time series feature vector; The migration instruction generation module is used to generate migration instructions based on the overall activity score of the code repository and the storage level where the code repository is located. The migration execution module is used to perform migration on the code repository according to the migration instructions and update the metadata of the code repository; and to identify hot files according to the historical access logs of the code repository and load them into the cache of the target storage level to which the migration is to take place.

[0051] An optional embodiment of the present invention provides a server, characterized in that it includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the above-described method.

[0052] An optional embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program implements the above-described method when executed by a processor.

[0053] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for migrating between multi-level intelligent storage and intelligent hierarchies of massive open-source code, comprising the following steps: Construct a logically unified but physically heterogeneous multi-level storage architecture to store data of varying popularity using storage media with different performance levels; The business flow of the code hosting platform is collected on a code repository basis, and a time-series feature vector of the code repository is generated. The overall activity score of the code repository is calculated based on the time series feature vector. Based on the overall activity score of the code repository and the storage level where the code repository is located, a migration instruction is generated; Perform migration on the code repository according to the migration instructions and update the metadata of the code repository; And based on the historical access logs of the code repository, hot files are identified and loaded into the cache of the target storage level to which they are being migrated.

2. The method of claim 1, wherein, The multi-level storage architecture includes an extremely hot layer for storing highly active code data, a hot layer for storing active projects, a warm layer for storing stable projects, and a cold layer for storing code identified as zombie code.

3. The method according to claim 2, characterized in that, The extreme heat layer uses an NVMe SSD storage cluster and is configured with DRAM memory or Redis as a first-level high-speed cache; the heat layer uses a distributed SSD storage pool and employs erasure coding strategy to segment and encode data; The warm layer uses a high-density HDD disk array and enables LZ4 for real-time data compression; the cold layer uses an object storage system or Blu-ray storage library and employs a high compression ratio algorithm for deep data compression.

4. The method according to claim 1, 2, or 3, characterized in that, The method for generating the time series feature vector is as follows: Extract the timestamps of code commits, the change in lines of code per commit, and the dispersion of file changes within each time slice of the code repository as code activity dimension data; Extract the number of new issues, closed issues, comment interaction frequency, and average time from commit to merge for each time slice in the code repository as data for community collaboration. The data includes statistics on the growth rate of stars / forks of a code repository in each time slice, the number of downstream dependent projects, and whether the code repository is associated with unpatched high-risk security vulnerabilities, serving as data for influence and security dimensions. Normalize the code activity dimension data, community collaboration dimension data, and influence and security dimension data to generate feature vectors for the code repository in each time slice; arrange the feature vectors of the code repository in each time slice in order to generate the time series feature vector of the code repository.

5. The method according to claim 4, characterized in that, The method for calculating the overall activity score of the code repository is as follows: The code activity dimension data is processed using a time decay function to calculate the real-time popularity value of the code repository; Predict the predicted popularity value of the code repository based on its historical time series characteristics; Calculate the weighted value of security events in the code repository based on data from the impact and security dimensions; The overall activity score of the code repository is calculated by combining the real-time popularity value, the predicted popularity value, and the security event weighted value.

6. The method according to claim 1, characterized in that, The method for generating the migration instructions is as follows: If the overall activity score of the code repository is higher than the admission threshold of the storage layer above the current storage layer of the code repository, and the duration of the overall activity score exceeds the set anti-jitter window, then an upward migration instruction is generated. If the code repository is marked as a security incident, then an upward migration instruction is generated; If the overall activity score of the code repository is lower than the retention threshold of the current storage level of the code repository, and it is predicted that the code repository will not have an upward trend in the near future, a downward migration instruction will be generated.

7. The method according to claim 1 or 6, characterized in that, Before generating the migration instruction, check the remaining capacity of the target storage tier to which the migration is to be performed; If the target storage tier has insufficient capacity, the code repositories will be sorted in descending order of their overall activity scores. The code repositories with the lowest overall activity scores in the target storage tier will be forcibly moved out, and the resulting space will be allocated to code repositories with higher overall activity scores.

8. A multi-level intelligent storage and intelligent hierarchy migration system for massive open-source code, characterized in that, It includes a storage resource pool construction module, a feature extraction module, a scoring module, a migration instruction generation module, and a migration execution module; The storage resource pool construction module is used to construct a logically unified but physically heterogeneous multi-level storage architecture, which uses storage media with different performance to store data with different popularity. The feature extraction module is used to collect the business flow of the code hosting platform on a code repository basis and generate the time series feature vector of the code repository. The scoring module is used to calculate the overall activity score of the code repository based on the time series feature vector; The migration instruction generation module is used to generate migration instructions based on the overall activity score of the code repository and the storage level where the code repository is located. The migration execution module is used to perform migration on the code repository according to the migration instructions and update the metadata of the code repository; And based on the historical access logs of the code repository, hot files are identified and loaded into the cache of the target storage level to which they are being migrated.

9. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images