Distributed mirror image library management method

Abstract

The invention provides a distributed mirror image library management method, which belongs to the technical field of distributed storage, and comprises the following steps of: counting the number of times that a mirror image file is requested, identifying a hotspot mirror image and a non-hotspot mirror image, respectively determining a corresponding cache space and a cache strategy from a mirror image-configuration table, obtaining a cache result, and creating a mirror image library framework. The method comprises the following steps of: setting a fragmentation strategy about initial resources of a mirror image library, constructing the mirror image library, setting an access strategy according to a user access log, performing mode division on the access strategy to obtain an access mode set, further determining an access rule, optimizing an access path of the mirror image library based on the access rule, and determining a fault-tolerant strategy of the mirror image library based on an optimization result. A fault processing mechanism is set, fault management is carried out on the mirror image library, caching, storage and access strategies and a fault-tolerant mechanism are systematically integrated, the performance, availability and expandability of the distributed mirror image library are effectively improved, and the requirements of large-scale data storage and access are better met.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage, and particularly to a method for managing a distributed mirror library. Background Art

[0002] The distributed storage technology originated from the growing demand for large-scale data storage and access. With the explosive growth of data volume, single-machine storage can no longer meet the requirements, and the distributed storage technology came into being. There are various distributed storage systems, but there is a lack of a systematic integration solution for caching, storage, access policies, and fault tolerance mechanisms.

[0003] Therefore, the present invention provides a method for managing a distributed mirror library. Summary of the Invention

[0004] A method for managing a distributed mirror library provided by the present invention identifies hot data by analyzing the mirror access frequency, and formulates caching policies and storage policies for hot and non-hot data respectively, analyzes the user access pattern, optimizes the access path, designs a fault tolerance mechanism based on the optimization result to ensure the high availability of the system, systematically integrates caching, storage, access policies, and fault tolerance mechanisms, effectively improves the performance, availability, and scalability of the distributed mirror library, and better meets the requirements of large-scale data storage and access.

[0005] The present invention provides a method for managing a distributed mirror library, including: Step 1: Count the number of times each mirror file is requested, identify hot mirrors and non-hot mirrors, determine the cache space and corresponding cache policies of hot mirrors and non-hot mirrors from the mirror-configuration table respectively, and obtain the cache result; Step 2: Create a mirror library architecture based on the cache result, set a sharding policy for the initial resources of the mirror library, and then construct the mirror library to store hot mirrors and non-hot mirrors according to the sharding policy; Step 3: Set an access policy according to the user access log, divide the access policy into patterns to obtain an access pattern set, determine the access rule based on the access pattern set, and optimize the access path of the mirror library based on the access rule; Step 4: Determine the fault tolerance policy of the mirror library based on the optimization result, set a fault handling mechanism according to the fault tolerance policy, and perform fault management on the mirror library.

[0006] The present invention provides a method for managing a distributed mirror library, which counts the number of times each mirror file is requested, identifies hot mirrors and non-hot mirrors, determines the cache space and corresponding cache policies of hot mirrors and non-hot mirrors from the mirror-configuration table respectively, and obtains the cache result, including: Use a data analysis tool to count the number of requests for each mirror file in the access logs of the mirror repository, set a hot threshold based on the number of requests, and classify the mirror files into hot mirrors and non-hot mirrors; Obtain the uses corresponding to the hot mirrors and non-hot mirrors, obtain the configurations of the cache space and cache policy from the mirror-configuration table, cache the mirror files, and obtain the cache result.

[0007] The present invention provides a distributed mirror repository management method, which obtains the uses corresponding to the hot mirrors and non-hot mirrors, obtains the configurations of the cache space and cache policy from the mirror-configuration table, caches the mirror files, and obtains the cache result, including: Determine the basic cache space for the hot mirrors and non-hot mirrors according to the uses corresponding to the hot mirrors and non-hot mirrors and the file sizes; Determine the total cache space according to the available disk space and memory size of the server, and calculate the allocable cache space and buffer space for each mirror file according to the ratio of the total cache space to the basic cache space; Determine the cache policy for each mirror file from the mirror-configuration table in combination with the allocable cache space, and cache the mirror files in combination with the cache policy and the buffer space, thereby obtaining the cache result.

[0008] The present invention provides a distributed mirror repository management method, which creates a mirror repository framework based on the cache result, sets a sharding strategy for the initial resources of the mirror repository, and then constructs a mirror repository to store the hot mirrors and non-hot mirrors according to the sharding strategy, including: Conduct a requirements analysis on the cache result to obtain the storage requirements, select a storage technology according to the storage requirements, and formulate a data migration strategy according to the cache result and the storage technology; Construct a mirror repository architecture according to the data migration strategy and storage requirements. At the same time, formulate a first sharding strategy according to the hot characteristics of the hot mirrors and a second sharding strategy according to the non-hot characteristics of the non-hot mirrors; Determine the sharding situation of the initial resources of the mirror repository according to the first sharding strategy and the second sharding strategy; Create a mirror repository by integrating the mirror repository framework and the sharding situation, and store the hot mirrors and non-hot mirrors.

[0009] The present invention provides a distributed mirror repository management method, which constructs a mirror repository architecture according to the data migration strategy and storage requirements. At the same time, formulate a first sharding strategy according to the hot characteristics of the hot mirrors and a second sharding strategy according to the non-hot characteristics of the non-hot mirrors, including: Based on the data migration strategy and storage requirements, clarify the total requirements of the mirror repository, and then construct a mirror repository framework; Set the first rule according to the access frequency in the hot spot characteristics of the hot spot mirror, set the second rule according to the geographical distribution of users, set the third rule according to the load balance, and obtain the first sharding rule based on the first rule, the second rule and the third rule, and then determine the first sharding strategy; Set the fourth rule according to the capacity ratio of the non-hot spot mirror, set the fifth rule according to the non-hot spot access mode, set the sixth rule according to the life cycle, and obtain the second sharding rule by integrating the fourth rule, the fifth rule and the sixth rule, and then determine the second sharding strategy.

[0010] The present invention provides a distributed mirror library management method, which sets an access policy according to the user access log, divides the access policy into modes, obtains an access mode set, determines the access rule based on the access mode set, and optimizes the access path based on the access rule, including: Determine the user type based on the user access log, set the priority according to the user type, set the frequency limit policy according to the priority, and determine the access policy by integrating the priority and the frequency limit policy; Determine the access mode according to the access policy in combination with the user access log, obtain the access mode set, and analyze the access rules between the access modes based on the access mode set; Adjust the access policy based on the access rule to obtain an optimized policy, and use the optimized policy to optimize the access path.

[0011] The present invention provides a distributed mirror library management method, which determines the fault tolerance policy of the mirror library based on the optimization result, sets a fault handling mechanism according to the fault tolerance policy, and performs fault management on the mirror library, including: Analyze the fault tolerance requirements of the mirror library based on the result of the access path optimization and the mirror library architecture, and select a fault tolerance policy according to the result of the requirement analysis; Determine the mirror library parameter configuration according to the fault tolerance policy, and set the alarm threshold based on the configuration result to obtain a fault monitoring mechanism; Determine the faulty nodes in the mirror library according to the fault monitoring mechanism, migrate the faulty nodes, and obtain a fault transfer mechanism; Integrate the fault monitoring mechanism and the fault transfer mechanism to determine the fault handling mechanism, and perform fault management on the mirror library according to the fault handling mechanism.

[0012] The present invention provides a distributed mirror library management method, which determines the mirror library parameter configuration according to the fault tolerance policy, and sets the alarm threshold based on the configuration result to obtain a fault monitoring mechanism, including: , wherein, represents the alarm threshold; B represents the alarm threshold of the system in the ideal state; The weight coefficient representing the resource usage amount; The maximum value representing the resource usage amount; The dynamic function representing the current resource usage amount; The weight coefficient representing the fault tolerance; The maximum value representing the fault tolerance; The dynamic function representing the current fault tolerance; The weight coefficient representing the metric value; The critical value representing the metric value; The dynamic function representing the current metric value; C represents a fixed adjustment factor of the system; The function representing the dynamic environmental factors; The function representing the monitoring tool accuracy; Represents a certain moment within the time window T, which is the integration variable; t represents the current time; Compare the real-time data with the alarm threshold. If the real-time data exceeds the alarm threshold, trigger an alarm; The degree of exceeding determines the severity of the alarm and sets different processing strategies, thereby obtaining a fault monitoring mechanism.

[0013] Compared with the prior art, the beneficial effects of the present application are as follows: By analyzing the mirror access frequency to identify hot data, formulating caching strategies and storage strategies for hot and non-hot data respectively, analyzing the user access pattern, optimizing the access path, designing a fault tolerance mechanism based on the optimization results to ensure the high availability of the system, systematically integrating caching, storage, access strategies and the fault tolerance mechanism, effectively improving the performance, availability and scalability of the distributed mirror library, and better meeting the requirements of large-scale data storage and access.

[0014] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the written specification and the drawings.

[0015] The following will further describe the technical solution of the present invention in detail through the drawings and embodiments. Description of the Drawings

[0016] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a flowchart showing a distributed mirror library management method provided by an embodiment of the present invention. Detailed Embodiments

[0017] The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0018] Embodiment 1: An embodiment of the present invention provides a distributed mirror library management method, as Figure 1 shown, including: Step 1: Count the number of times each mirror file is requested, identify hot mirrors and non-hot mirrors, and respectively determine the cache space and corresponding cache policies for hot mirrors and non-hot mirrors from the mirror-configuration table to obtain a cache result; Step 2: Create a mirror library architecture based on the cache result, set a sharding policy for the initial resources of the mirror library, and then build a mirror library to store hot mirrors and non-hot mirrors according to the sharding policy; Step 3: Set an access policy according to the user access log, divide the access policy into modes to obtain an access mode set, determine the access rule based on the access mode set, and optimize the access path of the mirror library based on the access rule; Step 4: Determine the fault tolerance policy of the mirror library based on the optimization result, set a fault handling mechanism according to the fault tolerance policy, and perform fault management on the mirror library.

[0019] In this embodiment, a hot mirror refers to a mirror file with a high access frequency, usually a mirror frequently requested by users. For example, in the scenario: in a Docker mirror library, a certain base mirror (such as ubuntu:latest) is frequently pulled. Data: It is statistically found that ubuntu:latest has been requested 10,000 times in the past week. Characteristics: High access frequency, great impact on system performance.

[0020] In this embodiment, a non-hot mirror refers to a mirror file with a low access frequency, usually a mirror rarely requested by users. For example, in the scenario: in a Docker mirror library, a certain specific version of the mirror (such as myapp:v1.2.3) is rarely pulled. Data: It is statistically found that myapp:v1.2.3 has only been requested 10 times in the past week. Characteristics: Low access frequency, small impact on system performance.

[0021] In this embodiment, the cache policy refers to the rule for managing the cache space, which determines which mirrors should be cached and when to be replaced. For example, LRU (Least Recently Used): Cache the most recently accessed mirrors first and replace the least recently used mirrors; FIFO (First In First Out): Cache the earliest accessed mirrors first and replace the earliest cached mirrors; LFU (Least Frequently Used): Cache the mirrors with the highest access frequency first and replace the mirrors with the lowest access frequency.

[0022] In this embodiment, the cache result refers to the final determined mirror cache status according to the cache space and cache policy. For example, in a scenario, the hot mirror ubuntu:latest is cached in a 10GB high-speed storage, while the non-hot mirror myapp:v1.2.3 is not cached and remains stored in the low-speed storage. As a result, when a user accesses ubuntu:latest, it is directly pulled from the cache, resulting in a faster speed. When a user accesses myapp:v1.2.3, it needs to be pulled from the original storage, resulting in a slower speed.

[0023] In this embodiment, the access logs of the mirror repository are statistically analyzed by a data analysis tool. A hot threshold is set according to the number of requests to distinguish hot and non-hot mirrors. Then, the mirrors are cached according to the mirror usage, cache space, and policy in the configuration table.

[0024] In this embodiment, the mirror repository architecture is the overall design of the mirror repository, including the storage layer, network layer, control layer, etc. For example, in a three-layer architecture, the bottom layer is a distributed file system, the middle layer is a cache server, and the top layer is an access control layer. The hot characteristics are the features of hot mirrors, such as high access frequency and low update frequency. The non-hot characteristics are the features of non-hot mirrors, such as low access frequency, possibly high update frequency, or large capacity.

[0025] In this embodiment, the sharding strategy includes a first sharding strategy and a second sharding strategy. The mirror repository architecture is constructed according to the data migration strategy and storage requirements. For hot mirrors, the first sharding strategy is formulated based on access frequency, geographical distribution, and load balancing. For non-hot mirrors, the second sharding strategy is formulated based on capacity ratio, access pattern, and lifecycle.

[0026] In this embodiment, the cache results are analyzed to determine the storage requirements and select appropriate storage technologies. A data migration strategy is formulated, and a mirror repository architecture is constructed based on this strategy and storage requirements. Different sharding strategies are formulated according to the characteristics of hot and non-hot mirrors, and finally, the mirror repository resources are sharded, and the mirror storage is completed.

[0027] In this embodiment, the user access log is a record of users accessing the mirror repository, including information such as time, user ID, and accessed mirror file. For example, a log record may contain: timestamp: 2024-10-27 10:00:00, user ID: user123, accessed mirror: ubuntu-22.04.iso, status: successful.

[0028] In this embodiment, the access policy is an access control rule formulated by integrating the priority and frequency limit policies, which may involve a queuing mechanism to give priority to processing requests from high-priority users.

[0029] In this embodiment, the access pattern set is various access patterns analyzed from user access logs. For example, high-frequency access, low-frequency access, sequential access, random access, etc. For instance, the access pattern set may include: high-frequency access (specific image, accessed multiple times), low-frequency access (many images, few access times each), sequential access (accessing images in sequence from the image list), random access (accessing different images irregularly).

[0030] In this embodiment, the access rule is the regularity between access patterns analyzed from the access pattern set. For example, the frequency of accessing a certain image is higher during a specific time period, certain images are always accessed together, etc. For instance, the rules may include: the access frequency in the morning on weekdays is higher than on weekends; image A and image B are often accessed together; a specific user group prefers a specific type of image.

[0031] In this embodiment, optimizing the access path is to adjust the access path according to the optimization strategy. For example, optimizing DNS resolution, using CDN acceleration, etc. For instance, the optimizations may include: using CDN to cache commonly used images; optimizing database queries to reduce the database load; adopting a faster network connection method.

[0032] In this embodiment, the fault tolerance strategy is a specific technical solution selected to meet the fault tolerance requirements. For example, adopting distributed storage and performing multi-copy redundant backup of data; adopting load balancing technology to distribute access requests to multiple nodes; using a heartbeat detection mechanism to monitor the running status of nodes.

[0033] In this embodiment, the fault handling mechanism is a fault handling process and strategy formulated by integrating a fault monitoring mechanism and a fault transfer mechanism. For example, when a node failure is detected, a fault transfer is automatically triggered; after manual confirmation of the fault, a recovery operation is performed; system backups and disaster recovery drills are carried out regularly.

[0034] In this embodiment, fault management is the overall management of the image library faults, including fault prevention, fault detection, fault handling, and fault recovery, etc. For example, formulating a perfect fault handling plan; performing system maintenance and upgrades regularly; conducting statistical analysis on faults to improve system design and operation and maintenance processes.

[0035] In this embodiment, based on the access path optimization results and the image library architecture, analyze and select a fault tolerance strategy, determine parameter configurations and alarm thresholds, establish a fault monitoring mechanism, through fault monitoring, identify and migrate faulty nodes, form a fault transfer mechanism, and finally establish a complete fault handling mechanism to achieve the fault management of the image library.

[0036] The working principle and beneficial effects of the above technical solution are as follows: By analyzing the mirror access frequency to identify hot data, caching strategies and storage strategies are formulated for hot and non-hot data respectively, the user access pattern is analyzed, the access path is optimized, and a fault tolerance mechanism is designed based on the optimization results to ensure the high availability of the system. The caching, storage, access strategies and fault tolerance mechanism are systematically integrated, effectively improving the performance, availability and scalability of the distributed mirror library, and better meeting the requirements of large-scale data storage and access.

[0037] Embodiment 2: The embodiment of the present invention provides a method for managing a distributed mirror library, which counts the number of times each mirror file is requested, identifies hot mirrors and non-hot mirrors, determines the cache space and corresponding cache strategies for hot mirrors and non-hot mirrors respectively from the mirror-configuration table, and obtains the cache result, including: Use a data analysis tool to count the number of requests for each mirror file in the access log of the mirror library, set a hot threshold according to the number of requests, and divide the mirror files into hot mirrors and non-hot mirrors; Obtain the uses corresponding to the hot mirrors and non-hot mirrors, obtain the configuration of the cache space and cache strategy from the mirror-configuration table, cache the mirror files, and obtain the cache result.

[0038] In this embodiment, the data analysis tool is software or a platform for processing, analyzing and visualizing data. For example, use Pandas to read the log file and count the number of requests for each mirror file.

[0039] In this embodiment, the hot threshold is a standard for the number of accesses used to distinguish hot mirrors and non-hot mirrors. Mirrors exceeding this threshold are regarded as hot mirrors. For example, the setting method: According to historical data, set the 90th percentile of the number of requests as the hot threshold. Example value: If 90% of the mirror requests are less than 1,000 times, then set the hot threshold to 1,000 times. Application: Mirrors with a request count > 1,000 times are hot mirrors, otherwise they are non-hot mirrors.

[0040] In this embodiment, the mirror-configuration table is a mapping table that records the cache space and cache strategy configuration corresponding to each mirror file. For example, mirror name, cache space, cache strategy, ubuntu:latest 10GB LRU (Least Recently Used), myapp:v1.2.3 1GB FIFO (First In First Out).

[0041] In this embodiment, the cache space refers to the storage capacity allocated to the mirror file, which is used to cache hot mirrors to improve the access speed. For example, scenario: Allocate 10GB of cache space for the hot mirror ubuntu:latest. Use: Store frequently accessed mirrors in the cache to reduce the pull time.

[0042] The working principle and beneficial effects of the above technical solution are as follows: By using a data analysis tool to count the access logs of the image repository, setting a hot threshold according to the number of requests, distinguishing hot and non-hot images, and caching the images according to the image usage and the cache space and policy in the configuration table, the access speed of hot images is improved, the server load is reduced, the resource utilization rate is optimized, and the overall system performance is enhanced.

[0043] Embodiment 3: An embodiment of the present invention provides a distributed image repository management method, which obtains the usages corresponding to hot images and non-hot images, obtains the configurations of cache space and cache policy from the image-configuration table, caches the image files, and obtains a cache result, including: Determining the basic cache space for hot images and non-hot images according to the usages corresponding to hot images and non-hot images and the file sizes; Determining the total cache space according to the available disk space and memory size of the server, and calculating the allocable cache space and buffer space for each image file according to the ratio of the total cache space to the basic cache space; Determining the cache policy for each image file from the image-configuration table in combination with the allocable cache space, and caching the image files in combination with the cache policy and the buffer space, thereby obtaining a cache result.

[0044] In this embodiment, the basic cache space refers to the initial cache space allocated for an image file according to its usage and file size. It is the basis for cache allocation and is usually determined according to the hotness degree and file size of the image. For example, for a hot image: Usage: Base image (such as ubuntu:latest), frequently used, File size: 500MB, Basic cache space: 1GB (2 times the file size); for a non-hot image: Usage: Specific version of application image (such as myapp:v1.2.3), rarely used, File size: 200MB, Basic cache space: 200MB (the same as the file size).

[0045] In this embodiment, the total cache space refers to the total disk space or memory space available on the server for caching image files, which is determined by the hardware resources of the server. For example, Server available disk space: 100GB, Server available memory size: 16GB, Total cache space: 50GB (determined comprehensively according to the disk space and memory size).

[0046] In this embodiment, the allocable cache space refers to the cache space dynamically allocated for each mirror file according to the ratio of the total cache space and the basic cache space. The calculation method is to calculate the total sum of the basic cache spaces of all mirror files, and determine the allocable cache space for each mirror file according to the ratio of the total cache space to the total sum of the basic cache spaces. For example, the basic cache space of the hot mirror ubuntu:latest is 1GB, the basic cache space of the non-hot mirror myapp:v1.2.3 is 200MB, the total sum of the basic cache spaces is 1GB + 200MB = 1.2GB, the total cache space is 50GB, the ratio of the allocable cache space is 50GB / 1.2GB ≈ 41.67, and the allocable cache space is: for ubuntu:latest, 1GB × 41.67 ≈ 41.67GB; for myapp:v1.2.3, 200MB × 41.67 ≈ 8.33GB.

[0047] In this embodiment, the buffer space refers to the additional cache space reserved to cope with sudden traffic or temporary demands. It usually accounts for a part of the total cache space. For example, the total cache space is 50GB, the ratio of the buffer space is 10%, the size of the buffer space is 50GB × 10% = 5GB, and the actual allocable cache space is 50GB - 5GB = 45GB.

[0048] In this embodiment, the cache policy refers to the rules for managing the cache space, which determines which mirrors should be cached and when to be replaced. Common policies include: LRU (Least Recently Used): Cache the most recently accessed mirrors first and replace the least recently used mirrors; FIFO (First In First Out): Cache the earliest accessed mirrors first and replace the earliest cached mirrors; LFU (Least Frequently Used): Cache the mirrors with the highest access frequency first and replace the mirrors with the lowest access frequency. For example, for the hot mirror ubuntu:latest, the cache policy is LRU (Least Recently Used) because it is frequently accessed and requires quick response; for the non-hot mirror myapp:v1.2.3, the cache policy is FIFO (First In First Out) because its access frequency is low and its cache priority is relatively low.

[0049] In this embodiment, the cache result refers to the final determined mirror cache status according to the allocable cache space, buffer space, and cache policy. For example, for the hot mirror ubuntu:latest, the allocable cache space is 41.67GB, the cache policy is LRU, and the cache result is that the mirror is cached in the 41.67GB high-speed storage and managed using the LRU policy; for the non-hot mirror myapp:v1.2.3, the allocable cache space is 8.33GB, the cache policy is FIFO, and the cache result is that the mirror is cached in the 8.33GB low-speed storage and managed using the FIFO policy.

[0050] The working principle and beneficial effects of the above technical solution are as follows: Calculate the basic cache space according to the mirror usage and size, and then determine the total cache space, the allocable cache space and buffer space for each mirror in combination with the server resources. Determine the cache policy according to the mirror configuration table and the allocable space, and perform caching in combination with the buffer space. Dynamically allocate the cache space according to the resources and requirements, optimize the cache policy, improve the cache efficiency, reduce the storage cost, and enhance the system performance.

[0051] Embodiment 4: The embodiment of the present invention provides a distributed mirror library management method, which creates a mirror library framework based on the cache result, sets a sharding strategy for the initial resources of the mirror library, and then constructs a mirror library to store hot mirrors and non-hot mirrors according to the sharding strategy, including: Conduct a requirements analysis on the cache result to obtain the storage requirements, select a storage technology according to the storage requirements, and formulate a data migration strategy according to the cache result and the storage technology; Construct a mirror library architecture according to the data migration strategy and storage requirements. At the same time, formulate a first sharding strategy according to the hot characteristics of hot mirrors and a second sharding strategy according to the non-hot characteristics of non-hot mirrors; Determine the sharding situation of the initial resources of the mirror library according to the first sharding strategy and the second sharding strategy; Create a mirror library by integrating the mirror library framework and the sharding situation, and store hot mirrors and non-hot mirrors.

[0052] In this embodiment, the requirements analysis is to analyze the cache result to determine the requirements for the storage capacity, performance, reliability, etc. of the mirror library, including evaluating the number, size, access frequency, and future growth trend of hot and non-hot mirrors. For example, the requirements analysis report indicates that at least 100TB of storage space is required, it can withstand 1000 read requests per second, and ensure 99.99% availability. The storage space requirement is expected to increase by 50% in the next three years.

[0053] In this embodiment, the storage requirements are the specific storage capacity, performance, and reliability indicators obtained based on the results of the requirements analysis. For example, 100TB of storage space, the read and write speeds reach 1000 times and 500 times per second respectively, and the data loss rate is less than 10⁻ 6, Specifically, the storage capacity is usually determined based on the data growth trend, business requirements, and historical data volume. For example, by statistically analyzing the data generation volume over a past period of time, a data growth model is established, such as an exponential growth model, a linear growth model, etc., to predict future storage requirements. At the same time, considering the business development plan, such as the impact of new business functions, the increase in the number of users, etc. on the data volume. Use data management tools and the statistical functions built into the database system to count the size and quantity of different types of data (such as hot and non-hot mirrors), and update them regularly to master the real-time scale of the data. Utilize data analysis software, such as Excel, the data analysis libraries of Python (Pandas, Numpy, etc.), to analyze the historical data volume, draw trend charts, and visually observe the data growth trend, so as to predict future storage capacity requirements.

[0054] The performance metrics of the storage system are closely related to the characteristics of business applications. For applications with frequent read requests, it is necessary to ensure that the storage system can provide sufficiently high read performance to guarantee the response speed of the business. At the same time, considering factors such as the read-write ratio of the data and the concurrent access situation, the read-write performance metrics of the storage system are comprehensively determined. For example, online transaction systems usually have relatively high requirements for read-write performance, while data warehouse systems may have more prominent requirements for read performance. Tools such as Iometer and Fio are used to simulate different workloads to perform performance tests on the storage system, obtaining performance metrics such as the read-write speed and IOPS (number of input / output operations per second) of the storage system under different conditions, providing a reference basis for determining the performance metrics. Use the monitoring tools built into the storage system or third-party monitoring software to monitor the performance metrics of the storage system in real time, such as CPU utilization, disk I / O, network bandwidth, etc. Through the analysis of the monitoring data, understand the performance performance of the storage system during actual operation, identify performance bottlenecks, and thus reasonably adjust the performance metrics.

[0055] Reliability metrics are mainly determined based on the importance and availability requirements of the business for data. For critical business data, such as financial transaction data, medical data, etc., higher reliability is usually required to prevent serious impacts on the business caused by data loss or damage. Common reliability metrics include data persistence, availability, fault tolerance, etc. Data persistence is ensured through technologies such as data redundancy and error correction codes; availability is achieved by adopting redundant architectures, failover mechanisms, etc. Technologies such as data replication and distributed storage are used to store data in multiple different locations or devices to prevent data loss caused by single-point failures. For example, using RAID (Redundant Array of Independent Disks) technology, data is distributed across multiple disks and parity checking and other methods are used to achieve data redundancy and fault tolerance. The stored data is regularly verified to ensure data integrity. At the same time, a data recovery mechanism is established to quickly recover data when data is lost or damaged. For example, using data backup software, data is backed up according to a certain strategy and recovery tests are regularly conducted to ensure the availability of the backup data.

[0056] The establishment of the fault monitoring mechanism is based on the monitoring and analysis of the system operation status. By setting reasonable alarm thresholds, potential faults in the system can be detected in a timely manner. The determination of the alarm thresholds is usually comprehensively considered according to the system's historical operation data, business requirements, and industry standards. For example, for the alarm threshold of CPU utilization, the average CPU utilization of the system in the normal operation state is generally referred to, and combined with the business requirements for system performance to determine. Monitoring systems such as Zabbix and Prometheus are deployed to monitor real-time metrics such as the CPU utilization, disk I / O, and network bandwidth of nodes. The monitoring system can collect the performance data of the system and compare it with the set alarm thresholds, and send alarm messages when the metrics exceed the thresholds. Data analysis techniques and machine learning algorithms are used to analyze the monitoring data to discover abnormal patterns and potential faults in the system. For example, by establishing a baseline model of system performance and using anomaly detection algorithms to detect abnormal behaviors deviating from the baseline, alarm signals are sent in a timely manner.

[0057] In this embodiment, the storage technology is a technical solution for meeting storage requirements. For example, distributed file systems (such as Ceph, Hadoop Distributed File System (HDFS)) are suitable for large-scale storage and high availability.

[0058] In this embodiment, the data migration strategy is a method for migrating existing mirror data to a new storage system. For example, full migration migrates all data to the new system at one time, incremental migration migrates a part of the data first and then periodically migrates the incremental data, and rolling migration gradually migrates the data from the old system to the new system while ensuring system availability.

[0059] In this embodiment, the initial resource sharding situation is the allocation of resources in the initial state of the image library. For example, for hot images, 80% of the cache and 20% of the low-speed storage are allocated, and for non-hot images, 20% of the cache and 80% of the low-speed storage are allocated.

[0060] In this embodiment, the first sharding strategy (for hot images) is how to allocate hot images to different storage nodes in the image library. For example, sharding is performed based on the access frequency, and images with high access frequency are allocated to nodes with higher performance. Geographical location can be considered, and the images are copied to nodes closer to the users.

[0061] In this embodiment, the second sharding strategy (for non-hot images) is how to allocate non-hot images to different storage nodes in the image library. For example, sharding is performed based on the image size, and large-capacity images are allocated to nodes with larger capacity. The access pattern and lifecycle of the images can be considered, and images with low access frequency and long lifecycle are allocated to nodes with lower storage costs.

[0062] The working principle and beneficial effects of the above technical solution are as follows: Analyze the cache results, determine the storage requirements and select appropriate storage technologies, formulate a data migration strategy, construct an image library architecture based on this strategy and storage requirements, formulate different sharding strategies according to the characteristics of hot and non-hot images, finally determine the resource sharding of the image library, and complete the image storage, optimizing the image library architecture and resource allocation, and improving the storage efficiency and access speed.

[0063] Embodiment 5: The embodiment of the present invention provides a distributed image library management method, which constructs an image library architecture according to the data migration strategy and storage requirements. At the same time, a first sharding strategy is formulated according to the hot characteristics of hot images, and a second sharding strategy is formulated according to the non-hot characteristics of non-hot images, including: Based on the data migration strategy and storage requirements, clarify the total requirements of the image library, and then construct the image library framework; Set a first rule according to the access frequency in the hot characteristics of hot images, set a second rule according to the geographical distribution of users, set a third rule according to the load balance, and obtain a first sharding rule according to the first rule, second rule and third rule, and then determine the first sharding strategy; Set a fourth rule according to the capacity ratio of non-hot images, set a fifth rule according to the non-hot access method, set a sixth rule according to the lifecycle, and obtain a second sharding rule by synthesizing the fourth rule, fifth rule and sixth rule, and then determine the second sharding strategy.

[0064] In this embodiment, the overall requirements are for the storage capacity, performance (read / write speed, IOPS), reliability (uptime, data persistence), and scalability of the entire image repository. These are derived from the storage requirement analysis based on previous caching results. For example, the overall requirements might be 200TB of storage space, the ability to handle 2,000 IOPS operations per second, an uptime of 99.9%, and a design that can easily scale to 500TB within the next two years.

[0065] In this embodiment, the first rule (based on access frequency) is that images with a higher access frequency have a higher priority. For example, an image accessed more than 1,000 times a day is considered a high-frequency access image.

[0066] In this embodiment, the second rule (based on user geographical distribution) is to copy the image to regions closer to the users to speed up access. For example, copy the image to data centers in North America, Europe, and Asia to serve users in these regions.

[0067] In this embodiment, the third rule (based on load balancing) is to distribute the images across multiple storage nodes to prevent any single node from being overloaded. For example, the load balancing algorithm ensures that no single node processes more than 20% of the total request volume.

[0068] In this embodiment, load balancing is a technique that evenly distributes the workload across multiple servers or storage nodes.

[0069] In this embodiment, the geographical distribution of users is the geographical location of the users accessing the image repository.

[0070] In this embodiment, the access frequency is the frequency at which each image is accessed (e.g., the number of requests per day, the number of requests per hour).

[0071] In this embodiment, the first sharding rule is to combine the first three rules to determine the rule for how to shard hot images, which involves a weighted scoring system where access frequency, geographical location, and load balancing all contribute to the final score, thus determining the placement of the image.

[0072] In this embodiment, the fourth rule (based on capacity ratio) is that the size of the image determines its storage location. Larger images may require a separate storage pool. For example, images over 10GB are placed in the high-capacity storage layer.

[0073] In this embodiment, the fifth rule (based on access pattern) is that images with infrequent access may be archived or placed in slower storage. For example, images accessed less than once a week are moved to the cold storage layer.

[0074] In this embodiment, the sixth rule (based on the life cycle) is that older and less important images can be stored in different ways. For example, images that are more than six months old are moved to a storage with lower cost and slower access speed.

[0075] In this embodiment, the second sharding rule is to combine the fourth, fifth, and sixth rules to determine the rule for sharding non-hot images. This may involve a hierarchical storage method that places images in different storage layers according to the access pattern and life cycle.

[0076] In this embodiment, the life cycle is the duration for which an image is expected to be actively used.

[0077] In this embodiment, the access method, such as sequential access or random access of the image, will affect the choice of storage technology.

[0078] The working principle and beneficial effects of the above technical solution are as follows: The image library architecture is constructed according to the data migration strategy and storage requirements. For hot images, the first sharding strategy is formulated based on the access frequency, geographical distribution, and load balancing. For non-hot images, the second sharding strategy is formulated based on the capacity ratio, access pattern, and life cycle. Through multi-dimensional rules, the refined management of the image library resources is realized, the storage efficiency and access performance are optimized, and the operation and maintenance costs are reduced.

[0079] Embodiment 6: The embodiment of the present invention provides a distributed image library management method, which sets an access policy according to the user access log, divides the access policy into patterns to obtain an access pattern set, determines the access rule based on the access pattern set, and optimizes the access path based on the access rule, including: Determine the user type based on the user access log, set the priority according to the user type, set the frequency limit policy according to the priority, and determine the access policy by combining the priority and the frequency limit policy; Determine the access pattern according to the access policy combined with the user access log to obtain an access pattern set, and analyze the access rule between the access patterns based on the access pattern set; Adjust the access policy based on the access rule to obtain an optimized policy, and use the optimized policy to optimize the access path.

[0080] In this embodiment, the user type is to classify users according to user behavior and attributes, such as ordinary users, paid users, administrators, internal testers, etc.

[0081] In this embodiment, the priority is the access priority of different user types. Users with a higher priority have higher access rights and resource allocation. For example, paid users > ordinary users > internal testers.

[0082] In this embodiment, the frequency limit policy is to set limits on the access frequency for different user types. For example, how many requests are allowed per second / minute / hour. For instance, for paid users: 10 requests per second; for ordinary users: 2 requests per second; for internal testers: 1 request per second.

[0083] In this embodiment, the optimization policy is to adjust the access policy based on access patterns. For example, increasing the cache of specific images, optimizing network routing, etc. For instance, the optimization policy may include: increasing the cache of frequently accessed images; adjusting network bandwidth allocation to prioritize the access of high-priority users; preloading images that are often accessed together.

[0084] The working principle and beneficial effects of the above technical solutions are as follows: Based on user access logs, set the frequency limit policy according to user types and priorities, determine the access policy, analyze the access logs, obtain the access pattern set and access rules, and accordingly adjust the access policy, optimize the access path. By analyzing user behavior, optimize the access policy, improve access efficiency, enhance the user experience, and can effectively prevent malicious access.

[0085] Embodiment 7: The embodiment of the present invention provides a distributed image library management method. Based on the optimization results, determine the fault tolerance policy of the image library, set up a fault handling mechanism according to the fault tolerance policy, and perform fault management on the image library, including: Based on the results of access path optimization and the image library architecture, analyze the fault tolerance requirements of the image library, and select a fault tolerance policy according to the results of the requirements analysis; Determine the image library parameter configuration according to the fault tolerance policy, and set the alarm threshold based on the configuration results to obtain a fault monitoring mechanism; Determine the faulty nodes in the image library according to the fault monitoring mechanism, migrate the faulty nodes to obtain a fault transfer mechanism; Integrate the fault monitoring mechanism and the fault transfer mechanism to determine the fault handling mechanism, and perform fault management on the image library according to the fault handling mechanism.

[0086] In this embodiment, the fault tolerance requirements refer to the fault tolerance capabilities that the image library needs to possess to ensure availability and reliability, which depend on the architecture, importance, and business requirements of the image library. For example, the image library needs to ensure 99.99% availability, be able to continue providing services in the case of a single-node failure, and the data loss rate is less than one in a million.

[0087] In this embodiment, the results of the requirements analysis are a specific analysis of the fault tolerance requirements, including the possible types of faults, the impacts of the faults, and the fault tolerance measures to be taken. For example, the analysis results show that the image library may face problems such as network faults, disk faults, and server faults, and technologies such as redundant backup and load balancing need to be adopted to improve reliability.

[0088] In this embodiment, the mirror repository parameter configuration is for implementing a fault tolerance strategy, and parameters need to be set for the mirror repository. For example, set the number of data replicas to 3, set the load balancing algorithm to the round-robin algorithm, and set the heartbeat detection interval to 10 seconds.

[0089] In this embodiment, the fault monitoring mechanism is a mechanism for detecting faults in the mirror repository, including monitoring metrics, alert thresholds, and alert methods. For example, monitor metrics such as the CPU utilization rate, disk I / O, and network bandwidth of nodes; set the CPU utilization rate exceeding 90% as the alert threshold, and send email and SMS alerts; use a log analysis tool to identify error logs.

[0090] In this embodiment, a faulty node is a mirror repository node that has failed. For example, a disk of a certain storage node is damaged, resulting in the inability to access the mirror files stored on it.

[0091] In this embodiment, the fault transfer mechanism is a mechanism for migrating data or services on a faulty node to other normal nodes. For example, use automatic fault transfer technology to automatically copy the data on the faulty node to other nodes; perform manual intervention to migrate the services on the faulty node to standby nodes.

[0092] The working principle and beneficial effects of the above technical solution are as follows: Based on the access path optimization results and the mirror repository architecture, analyze and select a fault tolerance strategy, determine parameter configuration and alert thresholds, establish a fault monitoring mechanism, identify and migrate faulty nodes through fault monitoring, form a fault transfer mechanism, and finally establish a complete fault handling mechanism to achieve the fault management of the mirror repository, improve the reliability and availability of the mirror repository, and reduce the impact caused by faults.

[0093] Embodiment 8: An embodiment of the present invention provides a distributed mirror repository management method, which determines the mirror repository parameter configuration according to the fault tolerance strategy, and sets the alert threshold based on the configuration result to obtain a fault monitoring mechanism, including: , Among them, represents the alert threshold; B represents the alert threshold in the ideal state of the system; represents the weight coefficient of the resource usage; represents the maximum value of the resource usage; represents the dynamic function of the current resource usage; represents the weight coefficient of the fault tolerance; represents the maximum value of the fault tolerance; represents the dynamic function of the current fault tolerance; represents the weight coefficient of the metric value; Represents the critical value of the metric value; Represents the dynamic function of the current metric value; C represents a fixed adjustment factor of the system; Represents the function of the dynamic environmental factors; Represents the function of the monitoring tool accuracy; Represents a certain moment within the time window T and is the integration variable; t represents the current time; Compares the real-time data with the alarm threshold. If the real-time data exceeds the alarm threshold, an alarm is triggered; The degree of exceeding determines the severity of the alarm and sets different handling strategies, thereby obtaining the fault monitoring mechanism.

[0094] In this embodiment, the resource usage is the amount of resources currently used by the system, such as CPU utilization, memory occupancy, disk I / O, etc. For example, the CPU utilization is 85%, the memory occupancy is 90%, and the disk I / O is 1000 times per second.

[0095] In this embodiment, the fault tolerance is the degree of fault that the system can tolerate. For example, the system's ability to still operate normally after some component failures occur. This is usually related to redundant design. For example, the system is designed with dual-machine hot standby, so that the system can continue to operate even if one server fails; the database has made data copy backups, so that the data can be restored even if some disks are damaged.

[0096] In this embodiment, the metric value is a key metric reflecting the system's operating state, such as network latency, request response time, error rate, etc. For example, the average network latency is 20ms, the request response time is 100ms, and the error rate is 0.5%.

[0097] In this embodiment, the monitoring tool accuracy is the accuracy of the monitoring tool in measuring the metric value. For example, the accuracy of the monitoring tool is ±1%, which means that there may be an error of ±1% in the measured CPU utilization.

[0098] In this embodiment, the dynamic environmental factors are environmental factors that affect the system's operating state, such as network load, number of user requests, etc. These factors change over time. For example, the network peak load causes an increase in network latency; the number of user requests surges during holidays.

[0099] In this embodiment, the severity is determined based on the degree to which the real-time data exceeds the alarm threshold. For example, minor: the real-time data slightly exceeds the alarm threshold; severe: the real-time data far exceeds the alarm threshold and may cause the system to crash; urgent: the real-time data has seriously affected the system operation and immediate measures need to be taken.

[0100] In this embodiment, the processing strategy is to take different processing measures for alarms of different severities. For example, for minor alarms, record logs and observe the system status; for severe alarms, start a pre-defined plan, such as increasing system resources and restarting some services; for emergency alarms, perform manual intervention, such as contacting relevant personnel for handling and conducting emergency system maintenance.

[0101] The working principle and beneficial effects of the above technical solution are as follows: Calculate the dynamic alarm threshold through a formula, comprehensively consider factors such as resource usage, fault tolerance, metric values, environmental factors, and monitoring accuracy. Compare the real-time data with the dynamic threshold. If the threshold is exceeded, an alarm is triggered, and the severity of the alarm and the processing strategy are determined according to the degree of exceedance. Finally, a fault monitoring mechanism is formed, which improves the accuracy and flexibility of alarms, reduces false alarms, and enhances system stability.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. However, such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed image library management method, characterized in that Including: Step 1: Count the number of times each mirror file is requested, identify hot mirrors and non-hot mirrors, determine the cache space and corresponding cache policies for hot mirrors and non-hot mirrors respectively from the mirror-configuration table, and obtain the cache result; Step 2: Create a mirror library architecture based on the cache result, set a sharding policy for the initial resources of the mirror library, and then build the mirror library to store hot mirrors and non-hot mirrors according to the sharding policy; Step 3: Set an access policy according to the user access log, divide the access policy into modes to obtain an access mode set, determine the access rule according to the access mode set, and optimize the access path of the mirror library based on the access rule; Step 4: Determine the fault tolerance policy of the mirror library based on the optimization result, set a fault handling mechanism according to the fault tolerance policy, and perform fault management on the mirror library.

2. The distributed mirror library management method according to claim 1, characterized in that, Count the number of times each mirror file is requested, identify hot mirrors and non-hot mirrors, determine the cache space and corresponding cache policies for hot mirrors and non-hot mirrors respectively from the mirror-configuration table, and obtain the cache result, including: Use a data analysis tool to count the request times of each mirror file in the access log of the mirror library, set a hot threshold according to the request times, and divide the mirror files into hot mirrors and non-hot mirrors; Obtain the uses corresponding to hot mirrors and non-hot mirrors, obtain the configurations of cache space and cache policy from the mirror-configuration table, cache the mirror files, and obtain the cache result.

3. The distributed mirror library management method according to claim 2, wherein Obtain the uses corresponding to hot mirrors and non-hot mirrors, obtain the configurations of cache space and cache policy from the mirror-configuration table, cache the mirror files, and obtain the cache result, including: Determine the basic cache space for hot mirrors and non-hot mirrors according to the uses corresponding to hot mirrors and non-hot mirrors and the file size; Determine the total cache space according to the available disk space and memory size of the server, and calculate the allocable cache space and buffer space for each mirror file according to the ratio of the total cache space to the basic cache space; Determine the cache policy for each mirror file from the mirror-configuration table in combination with the allocable cache space, and cache the mirror files in combination with the cache policy and the buffer space, thereby obtaining the cache result.

4. A distributed mirror library management method according to claim 1, characterized in that, Create a mirror library architecture based on the cache result, set a sharding policy for the initial resources of the mirror library, and then build the mirror library to store hot mirrors and non-hot mirrors according to the sharding policy, including: Conduct a requirements analysis on the cache result to obtain the storage requirements, select a storage technology according to the storage requirements, and formulate a data migration policy according to the cache result and the storage technology; Build a mirror library architecture according to the data migration policy and storage requirements. At the same time, formulate a first sharding policy according to the hot characteristics of hot mirrors and a second sharding policy according to the non-hot characteristics of non-hot mirrors; Determine the sharding situation of the initial resources of the mirror library according to the first sharding policy and the second sharding policy; Create a mirror library by integrating the mirror library architecture and the sharding situation, and store hot mirrors and non-hot mirrors.

5. A distributed image library management method according to claim 4, characterized in that, Construct an image library architecture according to the data migration strategy and storage requirements. At the same time, formulate a first sharding strategy based on the hot characteristics of hot images and a second sharding strategy based on the non-hot characteristics of non-hot images, including: Based on the data migration strategy and storage requirements, clarify the overall requirements of the image library, and then construct the image library framework; Set a first rule according to the access frequency in the hot characteristics of hot images, a second rule according to the geographical distribution of users, a third rule according to load balancing, and obtain a first sharding rule based on the first rule, second rule, and third rule, and then determine the first sharding strategy; Set a fourth rule according to the capacity ratio of non-hot images, a fifth rule according to the non-hot access method, and a sixth rule according to the life cycle. Combine the fourth rule, fifth rule, and sixth rule to obtain a second sharding rule, and then determine the second sharding strategy.

6. A distributed image library management method according to claim 1, characterized in that, Set an access strategy according to the user access log, perform pattern division on the access strategy to obtain an access pattern set, determine the access pattern based on the access pattern set, and optimize the access path based on the access pattern, including: Determine the user type based on the user access log, set the priority according to the user type, set the frequency limit strategy according to the priority, and combine the priority and frequency limit strategy to determine the access strategy; Determine the access pattern according to the access strategy combined with the user access log to obtain an access pattern set, and analyze the access pattern between access patterns based on the access pattern set; Adjust the access strategy based on the access pattern to obtain an optimized strategy, and use the optimized strategy to optimize the access path.

7. A distributed image library management method according to claim 1, characterized in that Determine the fault tolerance strategy of the image library based on the optimization result, set a fault handling mechanism according to the fault tolerance strategy, and perform fault management on the image library, including: Analyze the fault tolerance requirements of the image library based on the result of access path optimization and the image library architecture, and select a fault tolerance strategy according to the result of requirement analysis; Determine the image library parameter configuration according to the fault tolerance strategy, and set the alarm threshold based on the configuration result to obtain a fault monitoring mechanism; Determine the faulty nodes in the image library according to the fault monitoring mechanism, and migrate the faulty nodes to obtain a fault transfer mechanism; Combine the fault monitoring mechanism and the fault transfer mechanism to determine the fault handling mechanism, and perform fault management on the image library according to the fault handling mechanism.

8. A distributed mirror library management method according to claim 7, characterized in that Determine the image library parameter configuration according to the fault tolerance strategy, and set the alarm threshold based on the configuration result to obtain a fault monitoring mechanism, including: ， Among them, represents the alarm threshold; B represents the alarm threshold of the system under ideal conditions; represents the weight coefficient of the resource usage; represents the maximum value of the resource usage; represents the dynamic function of the current resource usage; represents the weight coefficient of the fault tolerance; represents the maximum value of the fault tolerance; represents the dynamic function of the current fault tolerance; represents the weight coefficient of the metric value; represents the critical value of the metric value; represents the dynamic function of the current metric value; C represents a fixed adjustment factor of the system; represents the function of the dynamic environmental factor; represents the function of the monitoring tool accuracy; represents a certain moment within the time window T and is the integration variable; t represents the current time; Compare the real-time data with the alarm threshold. If the real-time data exceeds the alarm threshold, trigger an alarm; Determine the severity of the alarm according to the exceeding degree, and set different handling strategies to obtain a fault monitoring mechanism.

Images