Data management method and device in storage system, equipment, medium and product

Abstract

The invention discloses a data management method and device in a storage system, equipment, a medium and a product, which are applied to the technical field of storage and can more reasonably and effectively manage storage data and metadata thereof in the storage system and further reduce the management cost and management difficulty of the storage system.The method comprises the steps that to-be-stored data is determined; wherein the to-be-stored data comprises target data and target metadata corresponding to the target data; dividing each target metadata into hot metadata and cold metadata according to a preset division proportion and the access frequency of each target metadata; storing the hot metadata to a high-speed storage pool, and storing target data corresponding to the hot metadata to a low-speed storage pool; and storing the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data management method in a storage system, and also to a data management device, electronic equipment, non-volatile storage medium and computer program product in the storage system. Background Art

[0002] In a distributed storage system, the object data and metadata are usually separated. The object data is stored on the HDD (Hard Disk Drive) and the metadata is stored on the SSD (Solid State Drive). This design strikes a balance between performance and cost. The low cost of HDD is suitable for large-scale storage needs, especially in data scenarios with infrequent access. The high performance of SSD can improve the overall performance of the system through fast access, especially in the case of high concurrent access. However, with the rapid development of related technologies, the number of objects continues to expand, and their metadata also grows accordingly. Due to the small size of metadata, the huge metadata will make object storage difficult to manage, which will impose huge overhead on the distributed storage system, that is, it will continuously expand the cost and management difficulty of the distributed storage system.

[0003] Therefore, how to manage the storage data and its metadata in the storage system more reasonably and effectively and further reduce the management cost and difficulty of the storage system is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] An object of the present invention is to provide a data management method in a storage system, which can manage the storage data and its metadata in the storage system more reasonably and effectively, further reducing the management cost and difficulty of the storage system; another object of the present invention is to provide a data management device, electronic device, non-volatile storage medium and computer program product in a storage system, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present invention provides a data management method in a storage system, comprising:

[0006] Determine the data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data;

[0007] According to a preset division ratio and an access frequency of each target metadata, each target metadata is divided into hot metadata and cold metadata;

[0008] The hot metadata is stored in a high-speed storage pool, and the target data corresponding to the hot metadata is stored in a low-speed storage pool;

[0009] The cold metadata and the target data corresponding to the cold metadata are stored in the low-speed storage pool.

[0010] The step of storing the hot metadata in a high-speed storage pool includes:

[0011] Saving the hot metadata to a preset cache;

[0012] When the data storage amount in the preset cache reaches a preset threshold, the hot metadata is flushed from the preset cache to the high-speed storage pool.

[0013] Wherein, flushing the hot metadata from the preset cache to the high-speed storage pool includes:

[0014] In the preset cache, each of the hot metadata is classified according to the internal record information of the hot metadata to obtain each hot metadata category;

[0015] For each hot metadata category, linearly merge the index information of each hot metadata under the hot metadata category to construct a first index corresponding to the hot metadata category;

[0016] The hot metadata under each hot metadata category is flushed from the preset cache to the high-speed storage pool according to each first index.

[0017] The hot metadata are classified according to the internal record information of the hot metadata to obtain the categories of the hot metadata, including:

[0018] Extracting features from the internal record information of the hot metadata to obtain target features; the target features include local features and / or request combination features;

[0019] The hot metadata are classified according to the target features to obtain categories of the hot metadata.

[0020] Wherein, according to each of the first indexes, each of the hot metadata under each of the hot metadata categories is flushed from the preset cache to the high-speed storage pool, including:

[0021] For each hot metadata, generate a key-value pair using the metadata name of the hot metadata and the hot metadata;

[0022] The key-value pairs under the hot metadata categories are flushed from the preset cache to the high-speed storage pool according to the first indexes.

[0023] Wherein, the data management method in the storage system further includes:

[0024] Determine a first query index according to the first data query instruction;

[0025] Determining a target key-value pair corresponding to the first query index in the high-speed storage pool;

[0026] Determine target hot metadata according to the target key-value pair;

[0027] The target query data corresponding to the target hot metadata is obtained by querying in the low-speed storage pool.

[0028] After the hot metadata is flushed from the preset cache to the high-speed storage pool, the method further includes:

[0029] All the hot metadata in the preset cache are deleted.

[0030] The step of storing the cold metadata and the target data corresponding to the cold metadata in the low-speed storage pool includes:

[0031] Generate a cold data group by correspondingly combining the cold metadata and the target data corresponding to the cold metadata;

[0032] Each of the cold data groups is stored in the low-speed storage pool.

[0033] The step of storing each of the cold data groups in the low-speed storage pool includes:

[0034] For each cold data group, linearly merge the index information of the cold metadata in the cold data group and the index information of the target data corresponding to the cold metadata to construct a second index corresponding to the cold data group;

[0035] Each of the cold data groups is stored in the low-speed storage pool according to each of the second indexes.

[0036] Wherein, the data management method in the storage system further includes:

[0037] Determine a second query index according to the second data query instruction;

[0038] Determining a target cold data group corresponding to the second query index in the low-speed storage pool;

[0039] Target cold metadata and target query data corresponding to the target cold metadata are determined according to the target cold data group.

[0040] Wherein, the data management method in the storage system further includes:

[0041] In the high-speed storage pool, when the access frequency of the hot metadata is lower than a first threshold, migrating the hot metadata to the low-speed storage pool;

[0042] In the low-speed storage pool, when the access frequency of the cold metadata exceeds a second threshold, the cold metadata is migrated to the high-speed storage pool.

[0043] Migrating the hot metadata to the low-speed storage pool includes:

[0044] Locking the hot metadata to obtain locked metadata;

[0045] Migrating the locking metadata to the low-speed storage pool;

[0046] In the low-speed storage pool, the locked metadata is unlocked to obtain unlocked metadata.

[0047] The step of locking the hot metadata to obtain the locked metadata includes:

[0048] Applying for a target lock from a lock service node, and using the target lock to lock the hot metadata to obtain the locked metadata;

[0049] Correspondingly, in the low-speed storage pool, after the locked metadata is unlocked to obtain the unlocked metadata, the method further includes:

[0050] Return the target lock to the lock service node.

[0051] Wherein, in the low-speed storage pool, after unlocking the locked metadata to obtain unlocked metadata, the method further includes:

[0052] Performing consistency check on the unlocking metadata;

[0053] When the consistency check fails, an alarm is output.

[0054] Wherein, the data management method in the storage system further includes:

[0055] When the consistency check fails, data rollback is performed on the unlocking metadata to migrate the unlocking metadata back to the high-speed storage pool.

[0056] Among them, the high-speed storage pool is a solid-state hard disk storage pool; the low-speed storage pool is a mechanical hard disk storage pool.

[0057] In a second aspect, the present invention further discloses a data management device in a storage system, comprising:

[0058] A determination module, used to determine data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data;

[0059] A division module, used for dividing each target metadata into hot metadata and cold metadata according to a preset division ratio and an access frequency of each target metadata;

[0060] A first saving module, used to save the hot metadata to a high-speed storage pool, and save the target data corresponding to the hot metadata to a low-speed storage pool;

[0061] The second saving module is used to save the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool.

[0062] In a third aspect, the present invention further discloses an electronic device, comprising:

[0063] Memory for storing computer programs;

[0064] A processor is used to implement the steps of any of the data management methods in the storage system described above when executing the computer program.

[0065] In a fourth aspect, the present invention further discloses a non-volatile storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data management method in any of the storage systems described above.

[0066] In a fifth aspect, the present invention further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the data management method in any of the storage systems described above.

[0067] The present invention provides a data management method in a storage system, comprising: determining data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data; dividing each target metadata into hot metadata and cold metadata according to a preset division ratio and an access frequency of each target metadata; saving the hot metadata to a high-speed storage pool, and saving the target data corresponding to the hot metadata to a low-speed storage pool; and saving the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool.

[0068] By applying the technical solution provided by the present invention, for the target data currently to be stored and the target metadata corresponding to the target data, all the target metadata to be stored can be divided into hot metadata and cold metadata with reference to the preset hot and cold data division ratio and the access frequency of each target metadata in a specific time period. Since the hot metadata has a higher access frequency and the cold metadata has a lower access frequency, the hot metadata can be saved to the high-speed storage pool in the storage system, and the cold metadata can be saved to the low-speed storage pool in the storage system. At the same time, the target data corresponding to the hot metadata and the target data corresponding to the cold metadata can be saved together in the low-speed storage pool, thereby realizing the reasonable use of the internal storage space of the storage system, that is, realizing more reasonable and effective management of the storage data and its metadata in the storage system, further reducing the management cost and difficulty of the storage system. On this basis, the hot and cold division of metadata effectively reduces the amount of data stored in the high-speed storage pool, thereby effectively reducing the research and development cost of the storage system.

[0069] The data management device, electronic device, non-volatile storage medium and computer program product in the storage system provided by the present invention also have the above-mentioned technical effects, and the present invention will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present invention, the following briefly introduces the drawings required for describing the prior art and the embodiments of the present invention. Of course, the drawings related to the embodiments of the present invention described below are only part of the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present invention.

[0071] Figure 1 A schematic diagram of a flow chart of a data management method in a storage system provided by an embodiment of the present invention;

[0072] Figure 2 An overall architecture diagram of a distributed storage system provided by an embodiment of the present invention;

[0073] Figure 3 A schematic diagram of the principle of a data classification and merging method provided by an embodiment of the present invention;

[0074] Figure 4 A schematic diagram of the structure of a data management device in a storage system provided by an embodiment of the present invention;

[0075] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The core of the present invention is to provide a data management method in a storage system, which can manage the storage data and its metadata in the storage system more reasonably and effectively, further reducing the management cost and difficulty of the storage system; another core of the present invention is to provide a data management device, electronic device, non-volatile storage medium and computer program product in a storage system, all of which have the above-mentioned beneficial effects.

[0077] In order to describe the technical solutions in the embodiments of the present invention more clearly and completely, the technical solutions in the embodiments of the present invention will be introduced below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] An embodiment of the present invention provides a data management method in a storage system.

[0079] Please refer to Figure 1 , Figure 1 This is a flow chart of a data management method in a storage system provided by an embodiment of the present invention. The data management method in the storage system may include the following S101 to S104.

[0080] S101: Determine data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data.

[0081] This step aims to determine the data to be stored, which is the various types of data that need to be saved in the storage system, including target data and target metadata corresponding to the target data. For example, in a scenario where the client initiates an IO (Input / Output) request (specifically, an input request) to the storage system, the data to be stored input by the client can be received through the network management service. Here, the storage system can specifically be a distributed object storage system.

[0082] S102: Divide each target metadata into hot metadata and cold metadata according to a preset division ratio and an access frequency of each target metadata.

[0083] This step aims to achieve the hot and cold classification of target metadata. It can be understood that cold metadata is metadata with a low access frequency, and hot metadata is metadata with a low access frequency. Therefore, this process can be achieved by referring to the access frequency of each target metadata and the pre-set cold and hot division ratio.

[0084] In the specific implementation process, the access frequency of each target metadata within a preset time period is first counted, and each target metadata is sorted in order of access frequency from large to small / from small to large to obtain a metadata sequence; then, all target metadata in the metadata sequence are divided into cold metadata and hot metadata according to a preset division ratio.

[0085] It should be noted that the specific values ​​of the preset division ratio and the preset time period do not affect the implementation of the technical solution, and can be set by the technical personnel in combination with actual needs, and the present invention does not limit this. For example, in a possible implementation, the preset division ratio can be 7:3, that is, 7 / 10 of the target metadata is determined as hot metadata, and 3 / 10 of the target metadata is determined as cold metadata; the preset time period can be 60 seconds.

[0086] S103: The hot metadata is stored in a high-speed storage pool, and the target data corresponding to the hot metadata is stored in a low-speed storage pool.

[0087] This step aims to save the hot metadata and its target data, that is, save the hot metadata to the high-speed storage pool, and save the target data (hot data) corresponding to the hot metadata to the low-speed storage pool. In a possible implementation, the high-speed storage pool can be a solid-state hard disk storage pool (such as an NVmeSSD storage pool); the low-speed storage pool can be a mechanical hard disk storage pool (such as an HDD storage pool).

[0088] In one embodiment of the present invention, storing the hot metadata to a high-speed storage pool may include:

[0089] Save hot metadata to preset cache;

[0090] When the data storage amount in the preset cache reaches a preset threshold, the hot metadata is flushed from the preset cache to the high-speed storage pool.

[0091] The embodiment of the present invention provides a method for implementing the storage of hot metadata to a high-speed storage pool, which can be implemented based on a preset cache. Specifically, the hot metadata can be first saved to the preset cache, and when the amount of data storage in the preset cache reaches a certain threshold condition, the hot metadata in the preset cache can be batch-flushed to the high-speed storage pool to effectively avoid the problem of system instability caused by frequent data flushing operations. Similarly, the specific value of the preset threshold can be set by the technician according to the actual situation, and this application does not limit this.

[0092] Furthermore, after the hot metadata is flushed from the preset cache to the high-speed storage pool, the method may also include: deleting all hot metadata in the preset cache. Thus, the storage space of the preset cache can be effectively saved, unnecessary storage space occupation can be avoided, and sufficient storage space can be provided for the hot metadata in the subsequent batch of data to be stored.

[0093] Among them, flushing the hot metadata from the preset cache to the high-speed storage pool may include: in the preset cache, classifying each hot metadata according to the internal record information of the hot metadata to obtain each hot metadata category; for each hot metadata category, linearly merging the index information of each hot metadata under the hot metadata category to construct a first index corresponding to the hot metadata category; and flushing each hot metadata under each hot metadata category from the preset cache to the high-speed storage pool according to each first index.

[0094] The embodiment of the present invention provides a method for flushing hot metadata from a preset cache to a high-speed storage pool, which can effectively improve the storage space utilization of the high-speed storage pool and help to achieve faster and more efficient data query. Specifically, the hot metadata can be merged with reference to the hot metadata category, and a new hot metadata index (i.e., the first index) can be constructed for it to realize the flushing of hot metadata based on the new hot metadata index.

[0095] Among them, the classification of hot metadata can be realized by referring to the internal record information of hot metadata. For example, a certain type of metadata is used to record the index information of the bucket where the corresponding object is located; a certain type of metadata is used to record the user and bucket information of the corresponding object, such as object encryption, object tags, access object audit logs and users, bucket policy permissions, etc.; a certain type of metadata is used to record the extended permissions of the corresponding object, such as system extended attributes and custom extended attributes.

[0096] In a possible implementation, classifying each hot metadata according to the internal record information of the hot metadata to obtain each hot metadata category may include: extracting features from the internal record information of the hot metadata to obtain target features; the target features include local features and / or request combination features; classifying each hot metadata according to each target feature to obtain each hot metadata category. In other words, hot metadata classification may be achieved by extracting relevant features of the internal record information of the hot metadata. Here, the extracted features of the internal record information of the hot metadata may include:

[0097] (1) Locality characteristics of metadata: Locality usually refers to the relationship between the physical location and access pattern of an object and its metadata in a distributed storage system. In many application scenarios, object access is often local. For example, for Internet video stream processing applications, multiple segments of a specific video (object) are usually read or written continuously. By optimizing the metadata storage location of these segments, performance can be improved. Based on this, the locality of object metadata includes metadata indexing, object permission policy control, object multi-version control, and custom extended attributes, which are the three types of metadata listed above.

[0098] (2) Metadata request combination characteristics: In the object storage operation requests Get, Put, List, Delete and Head, the Get operation is usually combined with the Head operation. That is, when downloading an object, a Head request is first made to the storage to query whether the object exists. If it exists, a Get request is made. Put and List operations are combined. That is, when uploading an object, a List request is first made to determine whether the object exists. If it does not exist, an upload operation is performed.

[0099] In a possible implementation, the hot metadata under each hot metadata category is flushed from the preset cache to the high-speed storage pool according to each first index, which may include: for each hot metadata, generating a key-value pair using the metadata name and the hot metadata of the hot metadata; and flushing each key-value pair under each hot metadata category from the preset cache to the high-speed storage pool according to each first index. In other words, the hot metadata can be saved in the high-speed storage pool in the form of a key-value pair, wherein the name of the hot metadata is used as the key value, and the hot metadata is used as the value value, forming a key-value pair to realize the storage of the hot metadata in the high-speed storage pool.

[0100] Based on the above embodiments, the data management method in the storage system provided by the embodiment of the present invention may further include:

[0101] Determine a first query index according to the first data query instruction;

[0102] Determining a target key-value pair corresponding to the first query index in the high-speed storage pool;

[0103] Determine target hot metadata according to the target key-value pair;

[0104] The target query data corresponding to the target hot metadata is obtained by querying in the low-speed storage pool.

[0105] The embodiment of the present invention provides a data query method based on a first index. Obviously, the index information to be queried can be determined according to the query instruction, the key-value pair to be queried can be determined according to the index to be queried, the metadata to be queried can be determined according to the key-value pair to be queried, and the data to be queried can be determined according to the metadata to be queried.

[0106] S104: Save the cold metadata and the target data corresponding to the cold metadata to a low-speed storage pool.

[0107] This step aims to save the cold metadata and its target data (cold data), that is, to save both to a low-speed storage pool.

[0108] In one embodiment of the present invention, storing the cold metadata and the target data corresponding to the cold metadata in a low-speed storage pool may include:

[0109] Generate a cold data group by corresponding the cold metadata and the target data corresponding to the cold metadata;

[0110] Each cold data group is saved in a low-speed storage pool.

[0111] An embodiment of the present invention provides a method for saving cold metadata and its target data (cold data) to a high-speed storage pool, that is, they can be saved together in a low-speed storage pool in the form of a "cold metadata-cold data" combination (cold data corresponds to cold metadata), which is more convenient for obtaining metadata and its data objects at the same time during subsequent data queries.

[0112] Among them, saving each cold data group to a low-speed storage pool may include: for each cold data group, linearly merging the index information of the cold metadata in the cold data group and the index information of the target data corresponding to the cold metadata to construct a second index corresponding to the cold data group; and saving each cold data group to a low-speed storage pool according to each second index.

[0113] The embodiment of the present invention provides a method for saving a cold data group to a low-speed storage pool, which can effectively improve the storage space utilization of the low-speed storage pool and help to achieve faster and more efficient data query. Specifically, for each cold data group, the index information of the cold data and its cold metadata can be directly linearly merged to achieve the construction of a new index (cold data group index, that is, the second index mentioned above), so as to achieve the preservation of the cold data group based on the new cold data group index.

[0114] Based on the above embodiments, the data management method in the storage system provided by the embodiment of the present invention may further include:

[0115] Determine a second query index according to the second data query instruction;

[0116] Determining a target cold data group corresponding to the second query index in the low-speed storage pool;

[0117] The target cold metadata and the target query data corresponding to the target cold metadata are determined according to the target cold data group.

[0118] The embodiment of the present invention provides a data query method based on a second index. Similar to the above-mentioned data query method based on the first index, it is obvious that the determination of the index information to be queried can be realized according to the query instruction, the determination of the data group to be queried can be realized according to the index to be queried, and the determination of the metadata to be queried and its data to be queried can be realized according to the data to be queried.

[0119] It can be seen that the data management method in the storage system provided by the embodiment of the present invention can refer to the preset hot and cold data division ratio and the access frequency of each target metadata in a specific time period for the target data currently required to be stored and the target metadata corresponding to the target data, and divide all the target metadata to be stored into hot metadata and cold metadata. Since hot metadata has a higher access frequency and cold metadata has a lower access frequency, the hot metadata can be saved to the high-speed storage pool in the storage system, and the cold metadata can be saved to the low-speed storage pool in the storage system. At the same time, the target data corresponding to the hot metadata and the target data corresponding to the cold metadata are also saved together in the low-speed storage pool, thereby realizing the reasonable use of the internal storage space of the storage system, that is, the storage data and its metadata are more reasonably and effectively managed in the storage system, further reducing the management cost and difficulty of the storage system. On this basis, the hot and cold division of metadata effectively reduces the amount of data stored in the high-speed storage pool, thereby effectively reducing the research and development cost of the storage system.

[0120] Based on the above embodiments:

[0121] In one embodiment of the present invention, the data management method in the storage system may further include:

[0122] In the high-speed storage pool, when the access frequency of the hot metadata is lower than a first threshold, the hot metadata is migrated to the low-speed storage pool;

[0123] In the low-speed storage pool, when the access frequency of the cold metadata exceeds a second threshold, the cold metadata is migrated to the high-speed storage pool.

[0124] The embodiment of the present invention provides a dynamic migration mechanism for metadata in a storage system. It is understandable that the access frequency of metadata in a storage system generally changes in real time, and the change in access frequency will inevitably lead to changes in the hot and cold types of metadata. Based on this, for each hot metadata in a high-speed storage pool, its access frequency can be monitored regularly / in real time. Once its access frequency is reduced and converted to cold metadata, it can be migrated to a low-speed storage pool; for each cold metadata in a low-speed storage pool, its access frequency can be monitored regularly / in real time. Once its access frequency is increased and converted to hot metadata, it can be migrated to a high-speed storage pool. It should be noted that the migration of hot and cold metadata will not cause changes in the storage location of its corresponding target data. Therefore, during the migration of hot and cold metadata, the storage location of the corresponding target data can be updated synchronously. In addition, the specific values ​​of the first threshold and the second threshold mentioned above do not affect the implementation of the present technical solution, and can be set by technical personnel according to actual conditions, and this application does not limit this.

[0125] In a possible implementation, migrating hot metadata to a low-speed storage pool may include: locking the hot metadata to obtain locked metadata; migrating the locked metadata to the low-speed storage pool; and unlocking the locked metadata in the low-speed storage pool to obtain unlocked metadata.

[0126] The embodiment of the present application provides an implementation method for migrating hot metadata to a low-speed storage pool, that is, a hot metadata migration method based on a locking mechanism, so as to effectively ensure the data consistency of the hot metadata during the migration process. In the implementation process, the hot metadata to be migrated can be locked first, and then the locked hot metadata (locked metadata) can be migrated to the low-speed storage pool, and finally unlocked. Of course, the locking mechanism is also applicable to the scenario of migrating cold metadata to a high-speed storage pool.

[0127] Among them, locking the hot metadata and obtaining the locked metadata may include: applying for a target lock from the lock service node, and using the target lock to lock the hot metadata to obtain the locked metadata; correspondingly, in the low-speed storage pool, after unlocking the locked metadata to obtain the unlocked metadata, it may also include: returning the target lock to the lock service node. In other words, the lock service node can be deployed in the storage system, and the lock service node can implement the distribution and recovery of the target lock. It should be noted that during the target lock distribution process, the lock service node needs to ensure that locking conflicts are avoided.

[0128] In addition, after unlocking the locked metadata in the low-speed storage pool to obtain the unlocked metadata, the process may further include: performing a consistency check on the unlocked metadata; and outputting an alarm prompt when the consistency check fails. Furthermore, when the consistency check fails, data rollback is performed on the unlocked metadata to migrate the unlocked metadata back to the high-speed storage pool.

[0129] That is to say, in order to further ensure the data consistency before and after data migration, a metadata consistency check mechanism and a data rollback mechanism when the check fails can be added after the metadata is unlocked. Specifically, when the consistency check of the unlocked metadata passes, it can be determined that the metadata migration is successful and no other operations are required; when the consistency check of the unlocked metadata fails, it can be determined that the metadata migration has failed, and at this time, an alarm operation and a metadata rollback operation can be performed.

[0130] The embodiment of the present invention takes a distributed storage system as an example and provides another data management method in a storage system.

[0131] First, please refer to Figure 2 , Figure 2 The overall architecture diagram of a distributed storage system provided by an embodiment of the present invention is based on scalability and fault tolerance. The design concept of the distributed storage system is to eliminate performance bottlenecks such as single-point metadata, adopt a distributed and decentralized design architecture, slice data and distribute them in the form of objects on multiple nodes, and ensure that the load and capacity of each node are balanced. Data access and management are performed through network connections, and a unified storage interface is provided. Since the distributed storage system adopts a decentralized design architecture, it is highly scalable, and the failure of a single storage node will not affect the unavailability of the entire distributed storage cluster; it supports dynamic expansion to add or delete storage nodes, and can automatically relocate data objects in the cluster to balance the storage space utilization between storage nodes.

[0132] Based on this distributed storage system, the distributed storage client initiates an object I / O request to the object storage gateway, and the gateway distributes the request to a unified, self-controlled, and scalable distributed storage consistency management system through an extensible hash calculation algorithm. The management system stores the object's index information, object metadata, and data in the storage pool. In the storage pool, the client's data is stored by slicing, which is fixed to 4MB objects by default. The objects are grouped, and the group ID is the value of the object Hash and the number of groups. Finally, it is stored in the underlying device NVmeSSD or HDD.

[0133] like Figure 2As shown in the figure, the distributed storage system architecture includes multiple components, such as object gateway service, unified, self-controlled, and scalable distributed storage consistency management system, storage pool, controllable, scalable, distributed data balanced placement algorithm, monitoring service cluster, etc. The functions of each component are introduced as follows:

[0134] 1. Object Gateway Service: Provides a unified storage interface to provide access interface services for distributed storage clients.

[0135] 2. Unified, self-controlled, and scalable distributed storage consistency management system: The core component of the distributed storage system provides storage services in the form of objects. In this component, data is stored as objects, each object has a unique identifier and related data. It is responsible for distributing objects on various nodes of the storage system and provides data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access.

[0136] 3. Storage pool: The distributed storage system consists of multiple storage nodes, each of which can contain multiple hard disks or storage devices. The storage pool divides the storage resources to form different pools to meet different storage needs. The data storage pool (low-speed storage pool) is composed of HDDs for cost considerations; the metadata storage pool (high-speed storage pool) is composed of NVmeSSDs to ensure efficient metadata access operations. Both data and metadata are stored in the form of objects on OSDs (object storage devices).

[0137] 4. Controllable, scalable, distributed data balanced placement algorithm: evenly distribute data objects across the storage nodes of the storage cluster to avoid data hot spots and improve system performance.

[0138] 5. Monitoring service cluster: responsible for monitoring the system status and configuration information to ensure the consistency and availability of the system.

[0139] Secondly, this embodiment proposes a predefined value θ for distinguishing metadata heat, which is used to distinguish the ratio of hot and cold metadata. The default value θ = 3 / 7, that is, 30% of the metadata is classified as hot metadata (metadata with a higher access frequency), and 70% of the metadata is classified as cold metadata (metadata with a lower access frequency). The predefined value θ can be flexibly adjusted according to the NVmeSSD space size of the distributed storage system. Among them, the statistics of metadata access frequency during the operation of storage IO services can be achieved by designing a time window, and the default is 60 seconds. On this basis, this embodiment provides different merging methods and writes to different storage media for hot and cold metadata. The merged hot metadata establishes a new index structure and is directly stored in the NVmeSSD storage pool; the cold metadata is merged with the data to establish a new index structure and is directly stored in the HDD storage pool. The data merging scheme is as follows:

[0140] 1. Hot metadata merging method: It should be noted that in a distributed object storage system, in order to effectively reduce the space occupied by object metadata on NvmeSSD, a distributed key-value storage key-value can be used to store object metadata, and RocksDB is used as a distributed key-value storage database. On this basis, since the frequently accessed object metadata is dispersed and persisted in different NvmeSSD storage locations in the RocksDB key-Value storage, that is, the hot metadata is stored non-linearly, this embodiment proposes the following merging scheme: First, after metadata heat statistics and classification, before writing to NVmeSSD, the storage location of the frequently accessed metadata, that is, the index information, is linearly merged to form a new metadata index structure; and then stored in NVmeSSD. Among them, the metadata that is frequently accessed can be merged by constructing a new key-value pair Key-Value pair, and then the Key-Value pair is linearly sorted. Among them, the object name in the object storage represents the key, that is, Key, and the associated high-access heat metadata is stored in Value. Therefore, when a client accesses a hot object, it first finds the primary index of the hot metadata, then finds the object metadata Key to be accessed in the primary index information, and locates the metadata Value through the key to obtain the object data to be accessed.

[0141] 2. Cold metadata and object data merging method: First, merge the metadata with the object data that is accessed infrequently and create an index structure; then store the merged result in the HDD. In this way, when the client makes an IO request for the metadata that is accessed infrequently, it can obtain the metadata and data of the object at the same time.

[0142] It is understandable that the purpose of data merging is to reduce the space occupied by object metadata on NvmeSSD, and at the same time, to reduce the number of metadata reconstructions during data balancing in fault scenarios, and to ensure efficient access to metadata within the distributed storage system; establishing a new index structure is to enable the merged metadata objects to achieve faster and more efficient data queries.

[0143] Please refer to Figure 3 , Figure 3 A schematic diagram of the principle of a data classification and merging method provided by an embodiment of the present invention. When a client initiates an object storage IO request, Object1, Object2, ..., ObjectN are received by the object gateway service and processed in the distributed storage system. Among them:

[0144] 1. Object metadata statistics and classification component: It collects statistics based on the characteristic information (locality characteristics and request combination characteristics) of client object storage IO requests, and identifies the data type of object metadata based on the statistical results for the next stage of processing.

[0145] The extraction of feature information can be implemented based on the internal record data of the object metadata. In this embodiment, the data types of the object metadata can be divided into the following three categories:

[0146] (1) Metadata A records the index information of the bucket where the object is located, that is, buckets.index contains the information of the bucket where the object is located.

[0147] (2) Metadata B records the user and bucket information of the object, such as object encryption, object tags, object access audit logs and users, bucket policy permissions, etc.

[0148] (3) Metadata C records the extended permissions of the object, such as system extended attributes, custom extended attributes, etc. Custom extended attributes allow users to add specific information, with "x-amz-meta-" as the prefix, such as x-amz-meta-author represents the name of the object creator, and x-amz-meta-description identifies the description information of the object; system extended attributes can include the object's data type Content-Type, such as image / jpeg, application / json, etc., and can also include the object's version ID, such as x-amz-version-id.

[0149] In addition, according to the type of the object metadata, the metadata may be sorted in the order of metadata A, metadata B, and metadata C from high priority to low priority.

[0150] It is worth noting that the statistics and classification process of the object metadata statistics and classification component can be carried out dynamically. When a failure or expansion occurs or a large amount of metadata is written, the distributed storage system will update the metadata according to dynamic business requests. Therefore, the classification of metadata will change accordingly, triggering the dynamic migration of metadata between different storage media, that is, high-speed storage pools and low-speed storage pools.

[0151] 2. High-speed pool component: Use NvmeSSD as a persistent storage device to store frequently accessed object metadata based on the processing results of the object metadata statistics and classification component. For example, merge the heat statistics Obj1.idx, Obj1.meta to Obj4.idx, Obj4.meta to form a new index metadata information of NObj1.idx and NObj1.meta.

[0152] 3. Low-speed pool component: HDD is used as the underlying storage device, and metadata of infrequently accessed objects is stored according to the processing results of the object metadata statistical classification component. For example, Obj1.idx, Obj1.meta, Obj1.data to Obj4.idx, Obj4.meta, Obj4.data are merged to store the metadata and data of the object together in the HDD.

[0153] Further, based on Figure 2 and Figure 3 As shown in the architecture diagram, this embodiment proposes the following storage IO process for data and metadata of distributed objects:

[0154] 1. When the client initiates an object storage write request, it is first managed in memory, including the hot metadata buffer, merging to generate a new object index table, and persistent storage. The memory space allocation method is:

[0155] ;

[0156] in, Indicates the number of objects written. Is the total memory usage, default It is 8GB and can be flexibly adjusted according to the scale of the distributed storage system. It is The memory usage of an object.

[0157] 2. For the written data objects, they are cached in the memory in the form of logs and flushed periodically. The cache is flushed because the memory space is limited.

[0158] 3. Store the heat metadata in the buffer cache structure and generate index information corresponding to it.

[0159] 4. Merge the objects in the buffer cache structure, generate new indexes and hot spot metadata and record them in the new object index table.

[0160] 5. Combine the metadata and data of infrequently accessed objects into large object blocks and write them into the distributed storage HDD storage pool.

[0161] 6. The merged heat metadata and object index are stored in the NVmeSSD storage pool in the form of Key-Value.

[0162] It can be seen that the logic of the storage IO process of data and metadata of distributed objects is as follows:

[0163] "for object data log written by the client do;

[0164] If the cache is refreshed then;

[0165] Store the heat metadata in the buffer cache structure;

[0166] Merge objects in the buffer cache structure, generate new indexes and hot spot metadata, and record them in the new object index table;

[0167] end if;

[0168] If the buffer structure capacity exceeds the threshold then;

[0169] while buffer is not empty do;

[0170] Merge metadata and data of low-frequency access objects into large object blocks and write them into the distributed storage HDD storage pool;

[0171] The merged heat metadata and object index are stored in the NVmeSSD storage pool in the form of Key-Value;

[0172] Delete the cache object in the buffer;

[0173] end while;

[0174] end if;

[0175] end for".

[0176] Finally, when a distributed storage system performs object IO storage, there are usually business fluctuations or changes. In this regard, this embodiment proposes a method for dynamically migrating object metadata. In the memory, the access frequency of the object metadata is obtained based on statistics. When the access frequency of the cold object metadata exceeds the predefined threshold, the object metadata is determined to be hot metadata and is migrated back from the HDD storage pool to the NVmeSSD storage pool. Conversely, hot metadata can also be dynamically migrated to the HDD storage pool based on the access frequency obtained from statistics. This process can be implemented based on the object metadata lock mechanism to ensure the consistency of the object metadata during migration. Based on this, the implementation process is as follows:

[0177] 1. Set up the object metadata lock mechanism in the distributed storage system: The implementation principle is to use cluster locks to protect access to metadata. The role of the lock is to ensure that during the migration process, only one process can read and write specific metadata. Specifically, the cluster lock method includes: setting the first node in the distributed storage system as a lock service node, and all requests that need to obtain locks need to apply to this lock service node; when a request is made for a lock, the lock service node will check whether there are other nodes currently holding the lock. If not, the lock will be assigned to the requesting node; if so, the request will be rejected. When the node holding the lock completes the operation, it will return the lock to the lock service node, and other nodes can apply for the lock again.

[0178] 2. Preparation before migration: Get the list of metadata to be migrated and lock the metadata to prevent other operations from modifying it.

[0179] 3. Perform migration: First, obtain the lock, then read the metadata from the NVmeSSD storage pool, lock it and write it to the HDD storage pool. After the migration is complete, update the storage location of the metadata.

[0180] 4. Unlock and consistency check: Unlock to allow other operations to access the migrated metadata. At the same time, perform consistency check on the metadata to ensure that the metadata can be accessed and used normally in the new storage pool.

[0181] 5. Error handling: During the migration process, if an error occurs, the migrated metadata can be rolled back to ensure metadata consistency and ensure that the failed migration does not affect the correctness of other concurrent operations.

[0182] Among them, during the dynamic migration process, the overhead of metadata migration, that is, the total migration time as follows:

[0183] ;

[0184] in, is the migration time, is the lock time (i.e., the time required for all metadata operations during migration), This is the time for consistency check and verification. Other operation time (the time required for applying other operations on metadata if there is no migration), is the maximum amount of metadata in the system, The amount of metadata being migrated.

[0185] It can be seen that the data management method in the storage system provided by the embodiment of the present invention can refer to the preset hot and cold data division ratio and the access frequency of each target metadata in a specific time period for the target data currently required to be stored and the target metadata corresponding to the target data, and divide all the target metadata to be stored into hot metadata and cold metadata. Since hot metadata has a higher access frequency and cold metadata has a lower access frequency, the hot metadata can be saved to the high-speed storage pool in the storage system, and the cold metadata can be saved to the low-speed storage pool in the storage system. At the same time, the target data corresponding to the hot metadata and the target data corresponding to the cold metadata are also saved together in the low-speed storage pool, thereby realizing the reasonable use of the internal storage space of the storage system, that is, the storage data and its metadata are more reasonably and effectively managed in the storage system, further reducing the management cost and difficulty of the storage system. On this basis, the hot and cold division of metadata effectively reduces the amount of data stored in the high-speed storage pool, thereby effectively reducing the research and development cost of the storage system.

[0186] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a data management device in a storage system provided by the present invention. The data management device in the storage system may include:

[0187] Determination module 1, used to determine the data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data;

[0188] A division module 2 is used to divide each target metadata into hot metadata and cold metadata according to a preset division ratio and an access frequency of each target metadata;

[0189] The first storage module 3 is used to store the hot metadata in a high-speed storage pool, and store the target data corresponding to the hot metadata in a low-speed storage pool;

[0190] The second saving module 4 is used to save the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool.

[0191] It can be seen that the data management device in the storage system provided by the embodiment of the present invention can refer to the preset hot and cold data division ratio and the access frequency of each target metadata in a specific time period for the target data currently to be stored and the target metadata corresponding to the target data, and divide all the target metadata to be stored into hot metadata and cold metadata. Since the hot metadata has a higher access frequency and the cold metadata has a lower access frequency, the hot metadata can be saved to the high-speed storage pool in the storage system, and the cold metadata can be saved to the low-speed storage pool in the storage system. At the same time, the target data corresponding to the hot metadata and the target data corresponding to the cold metadata are also saved to the low-speed storage pool, thereby realizing the reasonable use of the internal storage space of the storage system, that is, the storage data and its metadata are more reasonably and effectively managed in the storage system, further reducing the management cost and difficulty of the storage system. On this basis, the hot and cold division of metadata effectively reduces the amount of data stored in the high-speed storage pool, thereby effectively reducing the research and development cost of the storage system.

[0192] In one embodiment of the present invention, the first storage module 3 may include:

[0193] A cache unit, used for saving hot metadata to a preset cache;

[0194] The flushing unit is used to flush the hot metadata from the preset cache to the high-speed storage pool when the data storage amount in the preset cache reaches a preset threshold.

[0195] In one embodiment of the present invention, the lower brush unit may include:

[0196] A classification subunit, used to classify each hot metadata in a preset cache according to internal record information of the hot metadata to obtain each hot metadata category;

[0197] A merging subunit, for linearly merging the index information of each hot metadata under each hot metadata category, to construct a first index corresponding to the hot metadata category;

[0198] The flushing unit is used to flush each hot metadata under each hot metadata category from a preset cache to a high-speed storage pool according to each first index.

[0199] In one embodiment of the present invention, the above-mentioned classification subunit can be specifically used to extract features from the internal record information of the hot metadata to obtain target features; the target features include local features and / or request combination features; and each hot metadata is classified according to each target feature to obtain each hot metadata category.

[0200] In one embodiment of the present invention, the above-mentioned flushing unit can be specifically used to generate a key-value pair for each hot metadata using the metadata name and hot metadata of the hot metadata; and flush each key-value pair under each hot metadata category from the preset cache to the high-speed storage pool according to each first index.

[0201] In one embodiment of the present invention, the data management in the storage system may also include a first query module, which is used to determine a first query index according to a first data query instruction; determine a target key-value pair corresponding to the first query index in a high-speed storage pool; determine target hot metadata based on the target key-value pair; and query in a low-speed storage pool to obtain target query data corresponding to the target hot metadata.

[0202] In one embodiment of the present invention, the first saving module 3 may further include a deleting unit for deleting all hot metadata in the preset cache after flushing the hot metadata from the preset cache to the high-speed storage pool.

[0203] In one embodiment of the present invention, the second storage module 4 may include:

[0204] A combining unit, used for correspondingly generating a cold data group from the cold metadata and the target data corresponding to the cold metadata;

[0205] The storage unit is used to store each cold data group in a low-speed storage pool.

[0206] In one embodiment of the present invention, the above-mentioned storage unit can be specifically used to linearly merge the index information of the cold metadata in the cold data group and the index information of the target data corresponding to the cold metadata for each cold data group, so as to construct a second index corresponding to the cold data group; and save each cold data group to a low-speed storage pool according to each second index.

[0207] In one embodiment of the present invention, the data management in the storage system may also include a second query module, which is used to determine a second query index according to a second data query instruction; determine a target cold data group corresponding to the second query index in a low-speed storage pool; and determine target cold metadata and target query data corresponding to the target cold metadata according to the target cold data group.

[0208] In one embodiment of the present invention, the data management in the storage system may also include a migration module, which is used to migrate hot metadata to a low-speed storage pool in a high-speed storage pool when the access frequency of hot metadata is lower than a first threshold; in a low-speed storage pool, when the access frequency of cold metadata exceeds a second threshold, migrate the cold metadata to a high-speed storage pool.

[0209] In one embodiment of the present invention, the migration module may include:

[0210] A locking unit, used to lock the hot metadata and obtain the locked metadata;

[0211] A migration unit, used to migrate the locked metadata to a low-speed storage pool;

[0212] The unlocking unit is used to unlock the locked metadata in the low-speed storage pool to obtain the unlocking metadata.

[0213] In one embodiment of the present invention, the locking unit may be specifically used to apply for a target lock from a lock service node, and use the target lock to lock the hot metadata to obtain the locking metadata;

[0214] Correspondingly, the migration module may further include a lock returning unit, which is used to return the target lock to the lock service node after unlocking the locking metadata in the low-speed storage pool to obtain the unlocking metadata.

[0215] In one embodiment of the present invention, the migration module may further include a verification unit for performing consistency verification on the unlocked metadata after unlocking the locked metadata to obtain the unlocked metadata in the low-speed storage pool; and outputting an alarm prompt when the consistency verification fails.

[0216] In one embodiment of the present invention, the migration module may further include a migration unit, which is used to roll back the unlocking metadata when the consistency check fails, so as to migrate the unlocking metadata back to the high-speed storage pool.

[0217] In one embodiment of the present invention, the high-speed storage pool may be a solid-state hard disk storage pool; and the low-speed storage pool may be a mechanical hard disk storage pool.

[0218] For an introduction to the device provided in the embodiment of the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0219] An embodiment of the present invention provides an electronic device.

[0220] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided by the present invention, and the electronic device may include:

[0221] A memory 11, used for storing computer programs;

[0222] The processor 10 can implement the steps of any of the above-mentioned data management methods in the storage system when executing a computer program.

[0223] like Figure 5, which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0224] In the embodiment of the present invention, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0225] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of a method for managing data in a storage system.

[0226] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present invention, the memory 11 at least stores programs for implementing the following functions:

[0227] Determine the data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data;

[0228] According to a preset division ratio and the access frequency of each target metadata, each target metadata is divided into hot metadata and cold metadata;

[0229] The hot metadata is saved in a high-speed storage pool, and the target data corresponding to the hot metadata is saved in a low-speed storage pool;

[0230] The cold metadata and the target data corresponding to the cold metadata are saved in a low-speed storage pool.

[0231] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0232] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0233] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0234] Of course, it should be noted that Figure 5 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 5More or fewer components than shown, or combinations of certain components.

[0235] An embodiment of the present invention provides a non-volatile storage medium.

[0236] The non-volatile storage medium provided in the embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, the steps of the data management method in any one of the above storage systems can be implemented.

[0237] Among them, the non-volatile storage medium can be any available medium that can be stored by a computer or a data storage device such as a server or data center that includes one or more available media integrated. For example, it can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk) and other media that can store computer program codes.

[0238] For an introduction to the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0239] An embodiment of the present invention provides a computer program product.

[0240] The computer program product provided in the embodiment of the present invention includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of any of the above-mentioned data management methods in the storage system can be implemented.

[0241] Specifically, in the above embodiments, all or part of them can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product.

[0242] Among them, the computer program product may include one or more computer programs / instructions, and when the computer program / instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention may be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a non-volatile storage medium, or transmitted from one non-volatile storage medium to another non-volatile storage medium. For example, computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0243] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0244] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0245] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0246] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0247] The technical solution provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principle and implementation of the present invention, and the description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data management method in a storage system, characterized in that: include: Determine the data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data; According to a preset division ratio and an access frequency of each target metadata, each target metadata is divided into hot metadata and cold metadata; The hot metadata is stored in a high-speed storage pool, and the target data corresponding to the hot metadata is stored in a low-speed storage pool; The cold metadata and the target data corresponding to the cold metadata are stored in the low-speed storage pool.

2. The data management method in a storage system according to claim 1, characterized in that: Saving the hot metadata to a high-speed storage pool includes: Saving the hot metadata to a preset cache; When the data storage amount in the preset cache reaches a preset threshold, the hot metadata is flushed from the preset cache to the high-speed storage pool.

3. The data management method in a storage system according to claim 2, characterized in that: Flushing the hot metadata from the preset cache to the high-speed storage pool includes: In the preset cache, each of the hot metadata is classified according to the internal record information of the hot metadata to obtain each hot metadata category; For each hot metadata category, linearly merge the index information of each hot metadata under the hot metadata category to construct a first index corresponding to the hot metadata category; The hot metadata under each hot metadata category is flushed from the preset cache to the high-speed storage pool according to each first index.

4. The data management method in a storage system according to claim 3, characterized in that: Classifying each of the hot metadata according to the internal record information of the hot metadata to obtain each hot metadata category includes: Extracting features from the internal record information of the hot metadata to obtain target features; the target features include local features and / or request combination features; The hot metadata are classified according to the target features to obtain categories of the hot metadata.

5. The data management method in a storage system according to claim 3, characterized in that: According to each of the first indexes, each of the hot metadata under each of the hot metadata categories is flushed from the preset cache to the high-speed storage pool, including: For each hot metadata, generate a key-value pair using the metadata name of the hot metadata and the hot metadata; The key-value pairs under the hot metadata categories are flushed from the preset cache to the high-speed storage pool according to the first indexes.

6. The data management method in a storage system according to claim 5, characterized in that: Also includes: Determine a first query index according to the first data query instruction; Determining a target key-value pair corresponding to the first query index in the high-speed storage pool; Determine target hot metadata according to the target key-value pair; The target query data corresponding to the target hot metadata is obtained by querying in the low-speed storage pool.

7. The data management method in a storage system according to claim 2, characterized in that: After flushing the hot metadata from the preset cache to the high-speed storage pool, the method further includes: All the hot metadata in the preset cache are deleted.

8. The data management method in a storage system according to claim 1, characterized in that: Saving the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool includes: Generate a cold data group by correspondingly combining the cold metadata and the target data corresponding to the cold metadata; Each of the cold data groups is stored in the low-speed storage pool.

9. The data management method in a storage system according to claim 8, characterized in that: Saving each of the cold data groups to the low-speed storage pool includes: For each cold data group, linearly merge the index information of the cold metadata in the cold data group and the index information of the target data corresponding to the cold metadata to construct a second index corresponding to the cold data group; Each of the cold data groups is stored in the low-speed storage pool according to each of the second indexes.

10. The data management method in a storage system according to claim 9, characterized in that: Also includes: Determine a second query index according to the second data query instruction; Determining a target cold data group corresponding to the second query index in the low-speed storage pool; Target cold metadata and target query data corresponding to the target cold metadata are determined according to the target cold data group.

11. The data management method in a storage system according to any one of claims 1 to 10, characterized in that: Also includes: In the high-speed storage pool, when the access frequency of the hot metadata is lower than a first threshold, migrating the hot metadata to the low-speed storage pool; In the low-speed storage pool, when the access frequency of the cold metadata exceeds a second threshold, the cold metadata is migrated to the high-speed storage pool.

12. The data management method in a storage system according to claim 11, characterized in that: Migrating the hot metadata to the low-speed storage pool includes: Locking the hot metadata to obtain locked metadata; Migrating the locking metadata to the low-speed storage pool; In the low-speed storage pool, the locked metadata is unlocked to obtain unlocked metadata.

13. The data management method in a storage system according to claim 12, characterized in that: The hot metadata is locked to obtain the locked metadata, including: Applying for a target lock from a lock service node, and using the target lock to lock the hot metadata to obtain the locked metadata; Correspondingly, in the low-speed storage pool, after the locked metadata is unlocked to obtain the unlocked metadata, the method further includes: Return the target lock to the lock service node.

14. The data management method in a storage system according to claim 12, characterized in that: In the low-speed storage pool, after unlocking the locked metadata to obtain unlocked metadata, the method further includes: Performing consistency check on the unlocking metadata; When the consistency check fails, an alarm is output.

15. The data management method in a storage system according to claim 14, characterized in that: Also includes: When the consistency check fails, data rollback is performed on the unlocking metadata to migrate the unlocking metadata back to the high-speed storage pool.

16. The data management method in a storage system according to claim 1, characterized in that: The high-speed storage pool is a solid-state hard disk storage pool; the low-speed storage pool is a mechanical hard disk storage pool.

17. A data management device in a storage system, characterized in that: include: A determination module, used to determine data to be stored; wherein the data to be stored includes target data and target metadata corresponding to the target data; A division module, used for dividing each target metadata into hot metadata and cold metadata according to a preset division ratio and an access frequency of each target metadata; A first saving module, used to save the hot metadata to a high-speed storage pool, and save the target data corresponding to the hot metadata to a low-speed storage pool; The second saving module is used to save the cold metadata and the target data corresponding to the cold metadata to the low-speed storage pool.

18. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data management method in the storage system according to any one of claims 1 to 16 when executing the computer program.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data management method in the storage system according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data management method in the storage system according to any one of claims 1 to 16 are implemented.

Images