Magneto-photoelectric storage all-in-one machine based on mass data multi-medium grading migration model

Abstract

The invention discloses a magneto-photoelectric storage all-in-one machine based on a mass data multi-medium grading migration model, and belongs to the technical field of information storage. The hierarchical management system comprises an input unit used for obtaining position information and access information of to-be-managed data; the classification unit is used for selecting a classification algorithm suitable for the to-be-managed data from an algorithm library based on a preset strategy library and / or equipment knowledge library and the access information transmitted by the input unit; and the migration unit is used for migrating the to-be-managed data to a proper storage position based on the classification of the to-be-managed data by the classification unit. According to the hierarchical management system and method, the characteristics of different storage media can be better utilized, the overall resource utilization rate of the storage system is improved, the storage cost is reduced, and the access efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of information storage technology, and in particular to a magneto-optical storage integrated machine based on a massive data multi-media hierarchical migration model. Background Art

[0002] In today's context of explosive data growth, the management and storage of massive data has become a key issue facing enterprises and research institutions. As the scale of data expands, not only does the demand for storage capacity grow rapidly, but it is also necessary to consider how to efficiently store, manage and migrate data to meet the multiple needs of cost control, data access efficiency and data security. To this end, hierarchical storage and data migration models have become a hot topic of research. This research needs to face the following status quo:

[0003] (1) Data explosion: With the rapid development of technologies such as the Internet, big data, the Internet of Things (IoT), and artificial intelligence, massive amounts of data are continuously generated in various industries. These data include not only structured data, but also a large amount of unstructured data, such as images, videos, and sensor data.

[0004] (2) Storage cost: The traditional single storage architecture is difficult to cope with the situation where data types are diverse and the length of data life cycle varies. In particular, when faced with a large amount of low-access frequency data that needs to be stored for a long time, the storage cost problem is particularly prominent. Therefore, it is necessary to introduce a hierarchical storage solution to store data in layers according to different values, access frequencies and other characteristics to save resources.

[0005] (3) Diversification of storage media: The types of media available in modern storage systems are constantly increasing. Common types include high-performance but high-cost flash memory (SSD), moderately priced and large-capacity hard disks (HDD), and more suitable tapes and optical disks (AD or Blu-AD, or Blu-ray Disc, BD) for long-term archiving. Different storage media have different performance, cost, and reliability, which makes it possible to design an efficient hierarchical storage system.

[0006] (4) Data life cycle: The life cycle of most data is changing, gradually transitioning from high-frequency use to low-frequency access or even archiving. For data at different stages, rationally selecting storage solutions (such as migrating cold data to lower-cost storage media) becomes an important means to improve the efficiency of storage systems.

[0007] (5) Data migration requirements: Data storage tiering involves not only static data distribution, but also a dynamic migration process. As data access patterns change or storage media age and become obsolete, data migration becomes an important part of ensuring the sustainability of storage systems. Data migration needs to consider migration costs, migration time, data integrity, and other issues.

[0008] In order to overcome the above problems, Chinese patent CN113792772B discloses a method for identifying hot and cold data for hierarchical mixed storage of data. The method uses a data temperature prediction model based on LSTM to predict the temperature of the data; the temperature attribute of the data is used as the basis for identifying the hot and cold degree of the data, and the model assumes that the data with a lower temperature at present has a lower probability of being accessed in the future, so it is regarded as cold data, and the data with a higher temperature at present has a high probability of being accessed in the future, so it is regarded as hot data; the hot and cold degree of the data is defined as the high or low temperature of the data: the hot and cold degree of the data is equivalent to the high or low temperature of the data. This method uses a temperature model to measure the hot and cold degree of the data, with the number of article accesses, access type, timestamp of each access, and storage medium where the article is accessed as input, and the temperature corresponding to the article as output. The prediction result is only related to the number of accesses, and it is impossible to pre-define rules to adapt to the special requirements of specific scenarios.

[0009] US patent application US20240232140A1 discloses a hierarchical management method, including: obtaining corresponding file activity data, including corresponding file access data, corresponding relative file value data and corresponding past access level data of a group of corresponding files; the system determines the heat score of each file in the group according to the activity data of each file; the system sorts the respective files according to their respective heat scores to select subgroups of the respective files for re-layering between storage layers. Among them, for constructing a heat score from a file operation log, the heat score is calculated by adding values ​​representing various file operations in a weighted manner. The new heat score of the file is combined with the previous heat score of the file to obtain a heat score reflecting historical and recent operations. The heat score of the file can also be calculated based on the number of current operations (e.g., read, write), the assigned relative importance value, and the past heat score of a given file. In one implementation, a relatively simple heat score formula calculates the heat score by summing the input and output operations of the file in the current time period, and combines the weighted average of the existing / previous heat scores of the file to be evaluated. It also has the above-mentioned disadvantages.

[0010] In summary, the existing hierarchical management and migration technologies still have some technical problems and defects, such as:

[0011] (1) Low data migration efficiency: When faced with massive amounts of data, the data migration process may take a long time, especially since online migration requires uninterrupted services, further exacerbating the migration time and cost issues. The migration process may also affect system performance and cause access delays.

[0012] (2) Insufficient intelligence: Although some systems use machine learning and intelligent prediction to optimize storage tiering and migration, the intelligence level of existing technologies is still limited. In particular, when dealing with complex and diverse usage patterns, it is impossible to achieve high-precision real-time prediction and automated adjustment.

[0013] (3) Poor cross-platform compatibility: Data migration between different storage platforms (such as different cloud service providers or local data centers) often faces compatibility issues, including inconsistent data formats, access rights, and protocols, which makes cross-platform migration complex and costly.

[0014] (4) Complex storage hierarchy management: As the complexity of storage system stratification increases, management costs rise. Administrators need to adjust data storage policies at different levels, and automated tools are still insufficient when handling special scenarios, increasing the difficulty of operation and maintenance.

[0015] (5) Long delay in accessing cold data: Although the storage cost is reduced for cold data that has been migrated to a low-cost storage layer, it often results in long delays when access is needed, which cannot meet the need for rapid data recovery in some situations.

[0016] Therefore, the existing hierarchical management technology still urgently needs research and development and breakthroughs, needs to adapt to different application scenarios, and improve storage efficiency. Summary of the invention

[0017] In view of this, the main purpose of the present application is to provide a hierarchical management system and method, and an integrated optical storage device using the same, in order to at least partially solve the above-mentioned technical problems.

[0018] In order to achieve the above objectives, as a first aspect of the present application, a hierarchical management system is proposed, comprising:

[0019] An input unit, used to obtain location information and access information of the data to be managed;

[0020] A classification unit, configured to select a classification algorithm suitable for the data to be managed from an algorithm library based on a preset policy library and / or device knowledge library and access information transmitted by the input unit;

[0021] The migration unit migrates the data to a suitable storage location based on the classification of the data to be managed by the classification unit.

[0022] As a second aspect of the present application, a hierarchical management system is also proposed, comprising:

[0023] An input unit, used to obtain location information and access information of the data to be managed, data update rules in specific application scenarios, and / or a user-defined update cycle;

[0024] The autonomous decision machine automatically operates in response to external input, or in response to a user-defined update cycle, or in response to monitoring of the idle state of the device; the autonomous decision machine automatically controls the classification unit and the migration unit based on preset conditions based on the policy library and / or the device knowledge library, and the access information transmitted by the input unit, so as to adjust the data to be managed, or part of the system storage data, or the entire system storage data to the optimal storage reading state;

[0025] A classification unit, used to select a classification algorithm suitable for the data to be managed from an algorithm library under the control of the autonomous decision-making machine;

[0026] The migration unit is used to migrate the data to be managed to a suitable storage location under the control of the autonomous decision-making machine.

[0027] As a third aspect of the present application, an integrated optical storage machine is also proposed, which uses optical storage as a main storage medium and includes the hierarchical management system as described above.

[0028] As a fourth aspect of the present application, a hierarchical management method is also proposed, comprising the following steps:

[0029] Obtain location information and access information of the data to be managed;

[0030] Based on a preset policy library and / or knowledge library, and access information of the data to be managed, selecting a classification algorithm suitable for the data to be managed from an algorithm library;

[0031] Based on the classification of the data to be managed, the data is migrated to a suitable storage location.

[0032] As the fifth aspect of the present application, an electronic system using the above-mentioned integrated optical storage device as a storage device is also proposed. The electronic system is preferably an archive management system, a library management system, a traffic management system, a video surveillance system, etc.

[0033] Based on the above technical solutions, it can be seen that the hierarchical management system and method of the present application have at least one of the following beneficial effects compared with the prior art:

[0034] 1. Improve resource utilization: Through hierarchical storage, the characteristics of different storage media can be better utilized to improve the overall resource utilization of the storage system.

[0035] 2. Reduce storage costs: By migrating infrequently accessed data to lower-cost storage media, the company's spending on storage devices is reduced.

[0036] 3. Improve access efficiency: Reasonable hierarchical storage and data migration strategies can achieve the best balance between performance and cost without affecting data access requirements.

[0037] 4. Dealing with data growth pressure: The continuous growth of massive data has put forward higher requirements for storage systems. Research on hierarchical storage and migration models will help solve the challenges brought by data growth. This application has important application value in cloud computing, big data storage, enterprise data management and other fields, and also provides a theoretical basis for the design of future intelligent storage systems.

[0038] 5. From the perspective of technological development, this application can achieve:

[0039] (1) Improve model accuracy: The implementation of core technologies can improve prediction accuracy and reduce error rates.

[0040] (2) Optimizing resource utilization: Technology conversion methods make the use of data storage and computing resources more efficient and reduce costs.

[0041] (3) Enhance system intelligence: Through online learning and adaptive models, the system can respond to changes in business needs in a timely manner and improve flexibility and adaptability.

[0042] 6. From the perspective of market application, this application can achieve:

[0043] (1) Improve storage efficiency: Through the hierarchical migration algorithm, data can be reasonably allocated to different storage media according to the access frequency and importance of the data, reducing the waste of storage resources.

[0044] (2) Reduce costs: Migrating infrequently used cold data to low-cost storage media (such as optical storage) can significantly reduce overall storage costs.

[0045] (3) Improve access performance: Hot data is stored on high-performance storage media (such as SSDs), which improves data access speed and optimizes user experience.

[0046] (4) Enhanced data management flexibility: Real-time monitoring and dynamic adjustment of storage strategies to ensure optimal storage of data under different time periods and conditions.

[0047] (5) Support for massive data processing: Able to efficiently process and manage massive data and adapt to rapidly growing data needs.

[0048] (6) Optimize energy consumption: Reduce energy consumption by rationally allocating data storage and reducing the frequency of use of high-performance storage media.

[0049] (7) Improve data security: Distribute data storage across multiple storage media to reduce the risk of single point failures and enhance data security and reliability.

[0050] In summary, the solution of the present application can be applied in a variety of scenarios, especially in areas that require efficient data management and storage optimization, and plays an important role. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0052] Figure 1 is a flow chart of the hierarchical management method of the present application;

[0053] Figure 2 This is a structural framework diagram of a hierarchical management system according to an implementation mode of the present application;

[0054] Figure 3 This is a structural framework diagram of a hierarchical management system according to another embodiment of the present application;

[0055] Figure 4 This is the basic workflow of the data multimedia hierarchical migration model of Example 1 of the present application;

[0056] Figure 5 It is a schematic diagram of the key basic working steps of the data multimedia hierarchical migration model of Example 1 of the present application. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0058] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0059] In this application, some terms have the following meanings:

[0060] The optical storage all-in-one machine mainly uses optical storage, and can also include an integrated cabinet for magnetic storage (mechanical hard disk HDD) and / or electrical storage (solid state disk SDD and flash memory Flash), with fully automatic internal operation. Optical storage, for example, is a number of Blu-ray optical drives, the maximum parallel connection can reach 24 groups, the number of Blu-ray discs can reach 6720, so the maximum storage capacity of a single machine can reach 3.2PB.

[0061] Hot data refers to data that is frequently accessed and critical to business and applications. This data usually needs to be accessed and processed quickly and efficiently, so it needs to be stored on high-performance, low-latency storage devices such as SSD solid-state drives and memory.

[0062] Warm data refers to data that has a moderate access frequency and is of certain importance to business and applications. This data does not need to be accessed and processed as quickly as hot data, but it also needs to be stored and accessed reliably within a certain period of time. Therefore, it can be stored on low-cost, large-capacity storage devices, such as disk arrays.

[0063] Cold data refers to data that is less frequently accessed and less important to business and applications. This data usually needs to be stored for a long time, but does not need to be frequently accessed and processed, so it can be stored on storage devices with lower costs and larger capacity, such as optical disk libraries and tape libraries.

[0064] With the advent of the information age, the amount of data has exploded, the data types have become more and more complex, and the data usage has also varied greatly, which has led to increasing difficulty in storing and managing this data. How to improve the efficiency of data storage and reduce data reading and writing time under limited equipment is a technical issue that urgently needs to be considered. Existing data hierarchical storage and management technologies are based on simple calculations and predictions, and are limited by various parameters of the device itself. They do not consider the complexity of storage management and the diversity of application scenarios, resulting in relatively low storage efficiency and high hardware requirements.

[0065] After in-depth research, the inventor organized and optimized various algorithms and proposed a hierarchical management system that can handle both multi-media hierarchical storage of massive data and quick storage and management of small application scenarios. It has good mobility, a wide range of adaptability, and is efficient and reliable.

[0066] Specifically, Figure 2 As shown, the hierarchical management system of the present invention includes:

[0067] An input unit, used to obtain location information and access information of the data to be managed;

[0068] A classification unit, configured to select a classification algorithm suitable for the data to be managed from an algorithm library based on a preset policy library and / or device knowledge library and access information transmitted by the input unit;

[0069] The migration unit migrates the data to a suitable storage location based on the classification of the data to be managed by the classification unit.

[0070] The access information of the data to be managed includes information such as the number of accesses to the data to be managed, the access type, the timestamp of each access, and the access initiating device.

[0071] When the input unit obtains the location information and access information, it stores the obtained information in a database, thereby establishing an association between the storage location of the data to be managed, the data heat type and the access information. The data heat type includes hot data, warm data and / or cold data.

[0072] The policy library includes processing strategies for data to be managed in specific application scenarios. For example, surveillance video data can be set to be warm data within 7 days and stored as cold data after 7 days. Another example is the satellite cloud image data of the weather forecast system, which is hot data within 24 hours and cold data after 24 hours. Through the setting of the policy library, special regulations of different application scenarios can be flexibly matched to avoid the defect that the artificial intelligence prediction algorithm is limited to the training model and cannot be flexibly changed.

[0073] The policy library also includes a summary of classification experiences for some data to be managed. For example, when it is detected that the data to be managed is a system file and is not a system file that is accessed daily, it is usually set as cold data.

[0074] The policy library may also include, for example, various classification policies based on data access frequency, data life cycle, and automatic classification in the prior art.

[0075] The device knowledge base is, for example, a collection of call interfaces for different storage systems, and the corresponding call parameters, call drivers, etc. are all collected, thereby breaking the limitations of a single storage system. For example, the current cloud device manufacturers such as AWS, Google Cloud and Azure, their storage management systems are only compatible with their own devices, and when adopting hierarchical migration management, they cannot take into account all devices and design the best system solution; the hierarchical management system of the present invention breaks the technical barriers between these manufacturers, regards all storable spaces as a whole, and performs comprehensive storage optimization, thereby greatly improving storage efficiency and shortening the average data access time; and the hierarchical management system of the present invention is also compatible with local devices and cloud devices as a whole, and data access and calling are more flexible and efficient.

[0076] The algorithm library may include various existing classification cache or storage algorithms, such as LRU (Least Recently Used) algorithm and LFU (Least Frequently Used) algorithm, or LRU-K (Least Recently Used K) algorithm and FIFO (First In First Out) algorithm, etc., and may also include optimization algorithms designed by the inventors themselves. The algorithm library preferably includes all existing algorithms and the applicable conditions of various algorithms, so as to determine whether they are applicable before calling.

[0077] In addition, when calling different algorithms, simulation comparison can be performed in combination with the device knowledge base, and sorting can be performed based on the simulation results to select the top three best algorithms for selection.

[0078] The migration unit is associated with a hybrid storage architecture, such as optical disk, magnetic tape, HDD and / or SSD hybrid storage. Optical disks and magnetic tapes are suitable for storing archived data that needs to be stored stably and has a very low access frequency; HDDs (hard disk drives) and disk arrays are suitable for storing a large amount of low-cost, large-capacity data, which can be warm data that is accessed occasionally, or cold data that is accessed infrequently; and SSDs (solid-state drives) provide high-performance read and write speeds for storing hot data that is accessed frequently.

[0079] The migration unit may use online migration technology, which ensures that data is still available during the migration process to avoid service interruption due to data movement. Such technology usually uses a dual write mode (write-through caching) to maintain data consistency on both the source storage and the target storage during the migration process.

[0080] The migration unit may also use deduplication and compression technology, which can reduce storage requirements and migration time during the data migration process. Deduplication technology can identify and delete redundant data during the migration process, thereby reducing the amount of data that needs to be migrated.

[0081] The migration unit adopts object storage technology. Object storage is widely used in cloud storage systems to meet the storage needs of unstructured data. Its elastic architecture and scalability make it suitable for large-scale data hierarchical storage.

[0082] The migration unit may also use predictive data migration technology, for example. The migration unit uses a prediction model based on historical access data to determine in advance which data will become "cold data", thereby performing pre-migration to further improve system efficiency.

[0083] The migration unit may also use disaster recovery and data redundancy technology. In the data migration process, it is necessary to consider the disaster recovery and redundancy of the data to prevent data loss or damage. Thus, for example, multiple copies of the data may be stored to retain multiple copies of the data in different geographical locations to improve the reliability of the system and the availability of the data. Alternatively, cross-platform data migration may be used to enable data to be moved between different cloud services or storage platforms.

[0084] like Figure 3 As shown, another embodiment of the hierarchical management system of the present invention includes:

[0085] An input unit, used to obtain location information and access information of the data to be managed, data update rules in specific application scenarios, and / or a user-defined update cycle;

[0086] The autonomous decision-making machine automatically runs in response to external input, or based on a user-defined update cycle, or based on the monitoring of the idle state of the device; it is similar to a collection of conditional judgment statements, based on a policy library and / or a device knowledge base, and access information transmitted by an input unit, and monitors and responds to various states in turn, and automatically adjusts the data to be managed, or part of the stored data, or the entire system to the optimal working state based on preset conditions.

[0087] A classification unit, used to select a classification algorithm suitable for the data to be managed from an algorithm library under the control of the autonomous decision-making machine;

[0088] The migration unit is used to migrate the data to be managed to a suitable storage location under the control of the autonomous decision-making machine.

[0089] The access information of the data to be managed includes information such as the number of accesses to the data to be managed, the access type, the timestamp of each access, and the access initiating device.

[0090] When the input unit obtains the location information and access information, it stores the obtained information in a database, thereby establishing an association between the storage location of the data to be managed, the data heat type and the access information. The data heat type includes hot data, warm data and cold data.

[0091] Among them, the data update rules under the specific application scenario include, for example, but are not limited to, specific application scenario||data source or affiliation||data type||dehumidification period (time to become warm data)||deactivation period (time to become cold data), etc.

[0092] The user-defined update cycle refers to the update cycle set by the user, for example, it can be set to once a week, once every two weeks, once a month, once every six months, etc. After setting this update cycle, the system automatically sets the time based on the setting. When the corresponding time point arrives, the system starts the update program, evaluates all the data stored in the system, and then migrates it according to the evaluation results to optimize the data storage status of the entire system.

[0093] The autonomous decision machine can be designed, for example, with a known structure, including a preset task pool and a processing unit, wherein the task pool stores various instructions that can be executed by the processing unit. The processing unit loads different initial programs based on different triggering conditions. For example, when triggered by external input, the autonomous decision machine only processes the data to be managed involved in the external input, and sequentially mobilizes the classification instructions and migration instructions in the task pool to classify and migrate them. When the user-defined update cycle is triggered, or the monitoring of the idle state of the device is triggered, the autonomous decision machine first calls the scanning instruction for the system storage data to obtain the classification and storage location of the system storage data, or it can directly call the classification-storage location correspondence table stored in the corresponding database, directly read the latest classification and storage status, and then call the classification instructions and migration instructions in the task pool to classify and update them, as well as necessary migration processing.

[0094] The policy library includes processing strategies for data to be managed in specific application scenarios. For example, surveillance video data can be set to be warm data within 7 days and stored as cold data after 7 days. Another example is the satellite cloud image data of the weather forecast system, which is hot data within 24 hours and cold data after 24 hours. Through the setting of the policy library, special regulations of different application scenarios can be flexibly matched to avoid the defect that the artificial intelligence prediction algorithm is limited to the training model and cannot be flexibly changed.

[0095] The policy library also includes a summary of classification experiences for some data to be managed. For example, when it is detected that the data to be managed is a system file and is not a system file that is accessed daily, it is usually set as cold data.

[0096] The policy library may also include, for example, various classification policies based on data access frequency, data life cycle, and automatic classification in the prior art.

[0097] The device knowledge base is, for example, a collection of call interfaces for different storage systems, and the corresponding call parameters, call drivers, etc. are all collected, thereby breaking the limitations of a single storage system. For example, the storage management systems of cloud device manufacturers such as AWS, Google Cloud, and Azure are only compatible with their own devices. When adopting hierarchical migration management, it is impossible to take all devices into account and design the best system solution. The hierarchical management system of the present invention breaks the technical barriers between these manufacturers, regards all storable spaces as a whole, and performs comprehensive storage optimization, thereby greatly improving storage efficiency and shortening the average data access time; and the hierarchical management system of the present invention is also compatible with local devices and cloud devices as a whole, making data access and calling more flexible and efficient.

[0098] The algorithm library may include various existing classification cache or storage algorithms, such as LRU (Least Recently Used) algorithm and LFU (Least Frequently Used) algorithm, or LRU-K (Least Recently Used K) algorithm and FIFO (First In First Out) algorithm, etc., and may also include optimization algorithms designed by the inventors themselves. The algorithm library preferably includes all existing algorithms and the applicable conditions of various algorithms, so as to determine whether they are applicable before calling.

[0099] In addition, when calling different algorithms, simulation comparison can be performed in combination with the device knowledge base, and sorting can be performed based on the simulation results to select the top three best algorithms for selection.

[0100] The migration unit is associated with a hybrid storage architecture, such as optical disk, magnetic tape, HDD and / or SSD hybrid storage. Optical disks and magnetic tapes are suitable for storing archived data that needs to be stored stably and has a very low access frequency; HDDs (hard disk drives) and disk arrays are suitable for storing a large amount of low-cost, large-capacity data, which can be warm data that is accessed occasionally, or cold data that is accessed infrequently; and SSDs (solid-state drives) provide high-performance read and write speeds for storing hot data that is accessed frequently.

[0101] The migration unit may use online migration technology, which ensures that data is still available during the migration process to avoid service interruption due to data movement. Such technology usually uses a dual write mode (write-through caching) to maintain data consistency on both the source storage and the target storage during the migration process.

[0102] The migration unit may also use deduplication and compression technology, which can reduce storage requirements and migration time during the data migration process. Deduplication technology can identify and delete redundant data during the migration process, thereby reducing the amount of data that needs to be migrated.

[0103] The migration unit adopts object storage technology. Object storage is widely used in cloud storage systems to meet the storage needs of unstructured data. Its elastic architecture and scalability make it suitable for large-scale data hierarchical storage.

[0104] The migration unit may also use predictive data migration technology, for example. The migration unit uses a prediction model based on historical access data to determine in advance which data will become "cold data", thereby performing pre-migration to further improve system efficiency.

[0105] The migration unit may also use disaster recovery and data redundancy technology. In the data migration process, it is necessary to consider the disaster recovery and redundancy of the data to prevent data loss or damage. Thus, for example, multiple copies of the data may be stored to retain multiple copies of the data in different geographical locations to improve the reliability of the system and the availability of the data. Alternatively, cross-platform data migration may be used to enable data to be moved between different cloud services or storage platforms.

[0106] The present application also discloses an optical storage integrated machine suitable for a multi-media hierarchical migration model of massive data, which includes a storage system with optical storage as the main storage and magnetic storage and / or electrical storage as the auxiliary storage. The system is closed and managed, and operates independently. The core technology is a hierarchical storage migration model. According to the access frequency, life cycle and storage cost of the data, the data is dynamically allocated to storage media with different performance and cost through intelligent strategies, and automatically migrated when the data usage pattern changes. This model combines a multi-level storage architecture, data classification and migration algorithms, and intelligent prediction technology to ensure that storage resource utilization is optimized and storage costs are reduced while meeting performance requirements.

[0107] Specifically, the optical storage integrated machine of the present application is a storage device that is constructed using the hierarchical management system as described above and is mainly based on optical storage. The multi-media data hierarchical storage device can dynamically allocate data to storage media of different performance and cost through intelligent strategies according to the access frequency, life cycle and storage cost of the data, and automatically migrate when the data usage pattern changes. The intelligent migration model it adopts combines a multi-level storage architecture, data classification and migration algorithms, and intelligent prediction technology to ensure that storage resource utilization is optimized and storage costs are reduced while meeting performance requirements.

[0108] The optical storage integrated machine may include, but is not limited to, 4, 6, 8, 10, 12, 16, 20, or 24 sets of optical disc drives, and may include, but is not limited to, 4 to 6720 Blu-ray Discs (BD).

[0109] The optical storage integrated machine is, for example, a magneto-optical storage integrated machine, or a magneto-optical storage integrated machine.

[0110] like Figure 1 As shown, the present application also discloses a hierarchical management method, comprising the following steps:

[0111] Obtain location information and access information of the data to be managed;

[0112] Based on a preset policy library and / or knowledge library, and access information of the data to be managed, selecting a classification algorithm suitable for the data to be managed from an algorithm library;

[0113] Based on the classification of the data to be managed, the data is migrated to a suitable storage location.

[0114] The hierarchical management method is not executed every time data is input, but the classification and migration of all data is performed after the system has been running for a long period of time and there is enough data to help train and predict the classification model.

[0115] The hierarchical management method is performed regularly, for example, once every six months or once a year.

[0116] The hierarchical management method is executed when the system detects idleness, that is, it is executed irregularly to improve the overall storage and reading efficiency of the system.

[0117] The present application also discloses an electronic system that uses the hierarchical management system as described above as a storage device, and the electronic system is, for example, a file management system, a library management system, a traffic management system, a video surveillance system, etc.

[0118] The present application will be further described below through specific embodiments. It should be noted that the following embodiments are only for illustration and are not intended to limit the present application. Based on the embodiments shown below in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0119] Example 1

[0120] This embodiment is a hierarchical management system, in which optical disks (Blu-AD or BD), mechanical hard disks (HDD) and solid state disks (SSD) are used as storage media, optical disks (Blu-AD or BD) are used as storage media for cold data, mechanical hard disks (HDD) are used as storage media for warm data, and solid state disks (SSD) are used as storage media for hot data. The hierarchical management system adopts the multi-media data hierarchical storage migration intelligent model technology.

[0121] Intelligent Model of Multi-media Data Hierarchical Storage and Migration

[0122] Based on the access frequency, life cycle and storage cost of data, intelligent strategies are used to dynamically allocate data to storage media with different performance and costs, and automatically migrate when the data usage pattern changes. This model combines a multi-level storage architecture, data classification and migration algorithms, and intelligent prediction technology to ensure that storage resource utilization is optimized and storage costs are reduced while performance requirements are met.

[0123] like Figure 4 As shown, the basic workflow of the data multimedia hierarchical migration model of the present application includes the entire process of data from access log to classification, prediction, decision-making, optimization, and then to monitoring and maintenance.

[0124] like Figure 5 As shown, the key steps include:

[0125] Data collection: Get historical access logs.

[0126] Feature extraction: Extract features related to prediction, such as access frequency, time period, etc.

[0127] Data preprocessing: Clean and normalize the data to ensure the accuracy of the model.

[0128] Model selection: Choose an appropriate forecasting model, such as time series analysis, random forest, etc.

[0129] Model training: Train the selected model using historical data.

[0130] Prediction Output: Generates predictions of future access patterns.

[0131] Model evaluation: Evaluate the predictive performance of the model and make adjustments if necessary.

[0132] Therefore, this hierarchical management system can obtain a classification algorithm for the corresponding data type based on a trained prediction model (it can adopt an existing time series analysis model or a model set up within the present invention), thereby classifying and migrating all stored data, and optimizing the model based on data evaluation and verification after several time periods (for example, one month, half a year or one year) to improve classification accuracy and storage reading efficiency.

[0133] Example 2

[0134] The present embodiment is an optical storage integrated machine, which uses a Blu-ray Disc (BD) and a solid-state drive (SSD) as storage media, wherein the Blu-ray Disc (BD) is used as a storage medium for cold data, and the solid-state drive (SSD) is used as a storage medium for hot data.

[0135] The integrated optical storage machine in this embodiment uses an autonomous decision-making machine to respond to external input, or to a user-defined update cycle, or to automatically operate in response to monitoring of the idle state of the device. It does not require external interference and can achieve complete black box operation, automated operation, and autonomous optimization. The storage and reading efficiency is far superior to the integrated optical storage machines currently on the market.

[0136] Example 3

[0137] The specific scheme is the same as Example 1, except that the specific storage entity is an intelligent optical storage integrated machine. In the intelligent optical storage integrated machine, the management architecture process code of the hierarchical storage migration model is implemented, involving the design of multiple modules, such as data classification, hierarchical storage, data migration, monitoring and security management. The following is a simplified code example that uses Python to simulate these processes. We will simulate the data storage, migration, and data classification and management of hot, warm, cold, and archive layers.

[0138] (1) Data classification and hierarchical decision making

[0139] Determine the storage tier based on the access frequency and life cycle of the data.

[0140]

[0141] (2) Automated Data Migration

[0142] Simulates automatic migration of data from one storage tier to another based on access patterns and policies.

[0143]

[0144]

[0145] (3) Monitoring and intelligent optimization

[0146] Monitor data access frequency and use machine learning models to predict future data access patterns and optimize storage strategies.

[0147]

[0148] (4) Security and data integrity checks

[0149] Encrypt and verify the integrity of data during migration.

[0150]

[0151]

[0152] (5) Cross-platform compatibility and cloud migration

[0153] Simulate data migration across platforms and cloud services, compatible with multiple storage solutions.

[0154]

[0155]

[0156] (6) Backup and Disaster Recovery

[0157] Implement data backup and disaster recovery mechanism.

[0158]

[0159] The above code simulates the management process of the hierarchical storage migration model, including data classification, automated migration, monitoring and optimization, security verification, cross-platform migration, and backup and recovery. These modules can be expanded according to actual needs, such as introducing more complex migration strategies, machine learning models to optimize storage hierarchy management, etc.

[0160] It can be seen from the above embodiments that the hierarchical management system and method of the present application have the following advantages:

[0161] 1. From the perspective of technological development:

[0162] 1) Improve model accuracy: The implementation of core technologies can improve prediction accuracy and reduce error rates.

[0163] 2) Optimize resource utilization: Technology conversion methods make the use of data storage and computing resources more efficient and reduce costs.

[0164] 3) Enhance system intelligence: Through online learning and adaptive models, the system can respond to changes in business needs in a timely manner and improve flexibility and adaptability.

[0165] 2. From the perspective of market application:

[0166] 1) Improve storage efficiency: Through the hierarchical migration algorithm, data can be reasonably allocated to different storage media according to the access frequency and importance of the data, reducing the waste of storage resources.

[0167] 2) Reduce costs: Migrating infrequently used cold data to low-cost storage media (such as optical storage) can significantly reduce overall storage costs.

[0168] 3) Improve access performance: Hot data is stored on high-performance storage media (such as SSDs) to improve data access speed and optimize user experience.

[0169] 4) Enhance data management flexibility: Real-time monitoring and dynamic adjustment of storage strategies to ensure optimal storage of data in different time periods and conditions.

[0170] 5) Support massive data processing: Able to efficiently process and manage massive data and adapt to rapidly growing data needs.

[0171] 6) Optimize energy consumption: Reduce energy consumption by rationally allocating data storage and reducing the frequency of use of high-performance storage media.

[0172] 7) Improve data security: Distribute data storage in multiple storage media to reduce the risk of single point failure and enhance data security and reliability.

[0173] In summary, the technical solution created by this application can be applied in a variety of scenarios, especially in areas that require efficient data management and storage optimization, and plays an important role.

[0174] The above description is only a preferred embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.

Claims

1. A hierarchical management system, characterized in that: include: An input unit, used to obtain location information and access information of the data to be managed; A classification unit, configured to select a classification algorithm suitable for the data to be managed from an algorithm library based on a preset policy library and / or device knowledge library and access information transmitted by the input unit; The migration unit migrates the data to a suitable storage location based on the classification of the data to be managed by the classification unit.

2. A hierarchical management system, characterized in that: include: An input unit, used to obtain location information and access information of the data to be managed, data update rules in specific application scenarios, and / or a user-defined update cycle; The autonomous decision machine automatically operates in response to external input, or in response to a user-defined update cycle, or in response to monitoring of the idle state of the device; the autonomous decision machine automatically controls the classification unit and the migration unit based on preset conditions based on the policy library and / or the device knowledge library, and the access information transmitted by the input unit, so as to adjust the data to be managed, or part of the system storage data, or the entire system storage data to the optimal storage reading state; A classification unit, used to select a classification algorithm suitable for the data to be managed from an algorithm library under the control of the autonomous decision-making machine; The migration unit is used to migrate the data to be managed to a suitable storage location under the control of the autonomous decision-making machine.

3. The hierarchical management system according to claim 1 or 2, characterized in that: The access information of the data to be managed includes the number of accesses to the data to be managed, the access type, the timestamp of each access and / or the access initiating device information; and / or When the input unit obtains the location information and the access information, the input unit stores the obtained information in a database, thereby establishing an association between the storage location of the data to be managed, the data heat type and the access information; wherein the data heat type includes hot data, warm data and / or cold data; and / or The policy library includes processing policies for data to be managed in specific application scenarios and / or a summary of classification experience for data to be managed.

4. The hierarchical management system according to claim 1 or 2, characterized in that: The policy library includes various classification policies based on data access frequency, data life cycle, and automated classification; and / or The device knowledge base includes knowledge of calling interfaces of different storage systems and specific calling driver programs.

5. The hierarchical management system according to claim 1 or 2, characterized in that: The migration unit adopts online migration technology to ensure that data is still available during the migration process to avoid service interruption due to data movement; and / or The migration unit adopts a dual write mode; and / or The migration unit adopts deduplication and compression technology; and / or The migration unit adopts object storage technology; and / or The migration unit adopts a predictive data migration technology; and / or The migration unit also adopts disaster recovery and data redundancy technology.

6. The hierarchical management system according to claim 2, characterized in that: The autonomous decision-making machine includes a preset task pool and a processing unit, and the task pool stores various instructions that can be executed by the processing unit.

7. An integrated optical storage machine, which uses optical storage as a main storage medium, and includes a hierarchical management system as described in any one of claims 1 to 6.

8. The integrated optical storage device according to claim 15, characterized in that: The optical storage integrated machine includes 4, 6, 8, 10, 12, 16, 20 or 24 groups of optical disk drives.

9. A hierarchical management method, characterized in that: The steps include: Obtain location information and access information of the data to be managed; Based on a preset policy library and / or knowledge library, and access information of the data to be managed, selecting a classification algorithm suitable for the data to be managed from an algorithm library; Based on the classification of the data to be managed, the data is migrated to a suitable storage location.

10. An electronic system using the hierarchical management system according to any one of claims 1 to 6 as a storage device, wherein the electronic system is preferably a file management system, a library management system, a traffic management system or a video surveillance system.

Images