First storage cluster, data storage device and system

Abstract

This application provides a first storage cluster, a data storage device, and a system. The first storage cluster is used to connect to at least one second storage cluster. The storage rate of the first storage cluster is greater than that of the second storage cluster. The first storage cluster includes a shared directory, which is used to store at least shared data of the second storage cluster. The first storage cluster provides a unified access point for accessing shared data stored in the shared directory, non-shared data of the first storage cluster, and data stored in shared storage units of the second storage cluster. Data stored in shared storage units can be migrated to the shared directory. This solution, through a unified access point, supports clients to simultaneously access both the first and second storage clusters, thereby improving client read and write efficiency.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a first storage cluster, data storage device and system. Background Technology

[0002] In the field of distributed storage, servers used for storing data are generally divided into two categories: all-flash and hybrid flash. All-flash servers are equipped with solid-state drives (SSDs) using the Non-Volatile Memory Host Controller Interface (NVM Express, NVMe) protocol, which have higher access speeds but also higher costs. Hybrid flash servers, on the other hand, mainly use mechanical hard drives (HDDs), which have lower access speeds but also lower costs. Summary of the Invention

[0003] In view of this, this application provides a first storage cluster, data storage device and system to improve data access rate without significantly increasing cost.

[0004] To achieve the above objectives, this application provides the following technical solution:

[0005] In a first aspect, this application provides a first storage cluster for connecting to at least one second storage cluster; the storage rate of the first storage cluster is greater than the storage rate of the second storage cluster.

[0006] The first storage cluster includes a shared directory, which is used to store at least the shared data of the second storage cluster;

[0007] The first storage cluster provides a unified access point, which is used to access the shared data stored in the shared directory, the non-shared data of the first storage cluster, and the data stored in the shared storage units of the second storage cluster; the data stored in the shared storage units can be migrated to the shared directory.

[0008] A second aspect of this application provides a data storage device applied to a first storage cluster provided in any one of the first aspects of this application. The first storage cluster includes a shared directory, and the data stored in the shared directory includes shared data of the second storage cluster. The device includes:

[0009] The reading unit reads the data to be shared from the shared storage unit of the second storage cluster based on the shared information of the second storage cluster;

[0010] The writing unit is used to write the data to be shared into the shared directory to generate the corresponding shared data.

[0011] A third aspect of this application provides a data storage system, the storage system comprising a first storage cluster and at least one second storage cluster provided by any of the first aspects of this application.

[0012] The advantage of this solution is that it transfers the data originally stored in the second storage cluster to the shared directory of the first storage cluster, thus utilizing the higher access speed of the first storage cluster to quickly query this data and improve data access efficiency. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the architecture of a first storage cluster and a second storage cluster disclosed in an embodiment of this application;

[0015] Figure 2 This is a flowchart illustrating a data storage method disclosed in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram illustrating the transfer of data from a first storage cluster to a second storage cluster, as disclosed in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram illustrating a metadata storage method disclosed in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram illustrating the writing of data from a first storage cluster to a second storage cluster, as disclosed in an embodiment of this application.

[0019] Figure 6 This is a flowchart illustrating reading data from a second storage cluster, as disclosed in an embodiment of this application.

[0020] Figure 7 This is a schematic diagram of the structure of a data storage device disclosed in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To facilitate understanding of the technical solutions of the embodiments of this application, some terms that may be involved in this embodiment will be explained first.

[0023] Plog refers to the linear storage units provided by each device (such as a server) in a storage cluster.

[0024] Slog (Storage log) data blocks (also simply called data blocks) are two-layer encapsulation structures based on Plog. They serve as storage units for data in the data system, recording information such as plog information, data compression type, and garbage rate within the data block. In this embodiment, Slog data blocks can be considered as a specific storage format for data in the first and second storage clusters; that is, all data is stored in the first and second storage clusters in the form of Slog data blocks.

[0025] The technical solution of this embodiment can be applied to any distributed storage scenario, such as enterprise-level distributed storage (EDS) scenarios.

[0026] This embodiment provides a first storage cluster for connecting to at least one second storage cluster; the storage rate of the first storage cluster is greater than the storage rate of the second storage cluster.

[0027] The first storage cluster includes a shared directory, which is used to store shared data from the second storage cluster.

[0028] The first storage cluster provides a unified access point, which is used to access shared data stored in the shared directory, non-shared data of the first storage cluster, and data stored in the shared storage units of the second storage cluster; data stored in the shared storage units can be migrated to the shared directory.

[0029] Shared data refers to data written to a shared directory using the methods of any embodiment of this application. For example, data to be shared and data to be read written to the shared directory in subsequent embodiments are all considered shared data.

[0030] The beneficial effect of this embodiment is that the data originally stored in the second storage cluster is transferred to the shared directory of the first storage cluster, so that the higher access speed of the first storage cluster can be used to quickly query this data, thereby improving the data access efficiency.

[0031] Furthermore, in related technologies, when a second storage cluster is connected to a first storage cluster, the two types of storage clusters typically use independent interfaces. Users need to frequently switch systems to find data, which affects user experience, and multiple systems increase maintenance costs. With the solution in this embodiment, after the second storage cluster is connected to the first storage cluster, the first storage cluster provides a unified access point, supporting clients to access local directories (i.e., non-shared data in the first storage cluster), shared directories, and shared storage units in the second storage cluster managed by the first storage cluster, thereby improving client read and write efficiency.

[0032] See Figure 2 The first storage cluster in the above embodiment can execute the following data storage method to store the data of the second storage cluster into the shared directory.

[0033] S201, based on the shared information of the second storage cluster, read the data to be shared from the shared storage unit of the second storage cluster.

[0034] S202, write the data to be shared to the shared directory to generate the corresponding shared data.

[0035] The shared information of the second storage cluster can be any information that reflects the read and write requirements of the data in the shared storage unit of the second storage cluster. Through the shared information, the first storage cluster can identify the data that is expected to be frequently read and written and the data that is not expected to be frequently read and written in the data of the shared storage unit, and prioritize reading the former as the data to be shared.

[0036] A shared directory can be understood as a virtual directory in the first storage cluster used to manage data. The shared directory corresponds to the storage space of the storage resource pool of the first storage cluster. Writing data to be shared into the shared directory is equivalent to writing the data to be shared into the storage space in the storage resource pool corresponding to the shared directory.

[0037] The beneficial effect of this embodiment is that, considering the limited capacity of the first storage cluster, this solution can selectively store some data of the second storage cluster in the shared directory, which can meet the needs of fast access and avoid the first storage cluster from being overloaded due to too much data stored in the shared directory.

[0038] The first storage cluster can be an all-flash cluster consisting of multiple all-flash servers, and the second storage cluster can be a hybrid flash cluster consisting of multiple mixed flash servers, or a hybrid flash cluster consisting of at least one all-flash server and at least one mixed flash server.

[0039] Optionally, the shared information includes the last update interval of the data in the shared storage units of the second storage cluster;

[0040] Based on the shared information of the second storage cluster, the data to be shared in the shared storage unit of the second storage cluster is read, including:

[0041] Data to be shared is read sequentially from the shared storage units of the second storage cluster in ascending order of the last update interval. The last update interval represents the time interval between the last time the corresponding data was updated and the current time.

[0042] For example, if the last time a piece of data in the second storage cluster 1 was updated was at time T0, and the current time is T1, then the last update interval of this data is T1-T0.

[0043] Using the method described above for reading data to be shared, the first storage cluster can prioritize migrating data with a smaller last update interval from the second storage cluster to the shared directory when there is sufficient storage space in the shared directory.

[0044] Specifically, the first storage cluster can determine the last update interval of the data stored in each data block in the second storage cluster, and read the data in each data block one by one in ascending order of the last update interval and write it to the shared directory until the usable capacity of the shared directory is exhausted.

[0045] As an example, suppose that in the second storage cluster 1, the amount of data whose last update interval is less than 24 hours is exactly equal to the 100GB of usable capacity allocated to the first storage cluster. When migrating data to be shared to the first storage cluster, the 100GB of data to be shared in the second storage cluster 1 whose last update interval is less than 24 hours will be migrated to the first storage cluster first. Data whose last update interval is greater than or equal to 24 hours will not be migrated to the first storage cluster for the time being. When there is spare capacity allocated to the first storage cluster, the most recently updated data will be migrated to the first storage cluster in ascending order of last update interval.

[0046] The advantage of doing this is that a smaller last update interval indicates that the corresponding data has only recently been updated. This type of data is more likely to be read and written by the client recently. Prioritizing the migration of data with a smaller last update interval to the shared directory can respond to client requests more quickly when clients read and write this type of data, thus improving data read and write efficiency.

[0047] Optionally, the shared information includes the historical access frequency of data in the shared storage units of the second storage cluster;

[0048] Based on the shared information of the second storage cluster, the data to be shared in the shared storage unit of the second storage cluster is read, including:

[0049] Data to be shared is read sequentially from the shared storage units of the second storage cluster in descending order of historical access frequency.

[0050] Using the method described above for reading data to be shared, the first storage cluster can prioritize migrating data with high historical access frequency from the second storage cluster to the shared directory when there is sufficient storage space in the shared directory.

[0051] Specifically, the first storage cluster can determine the historical access frequency of the data stored in each data block in the second storage cluster, and read the data in each data block one by one in descending order of historical access frequency and write it to the shared directory until the usable capacity of the shared directory is exhausted.

[0052] Historical access frequency can be represented by the average number of visits per unit of time (one unit of time can be one minute) within a recent period (such as the last hour or the last 10 minutes).

[0053] As an example, when the migration module migrates data from the second storage cluster 1 to the first storage cluster, it arranges the data in the second storage cluster 1 in descending order of historical access frequency, and then selects the data with the highest historical access frequency one by one as the data to be shared, until the total amount of the selected data to be shared reaches the 100GB of usable capacity allocated to the first storage cluster. Then, this 100GB of data to be shared is migrated to the first storage cluster. After that, whenever there is available capacity, the migration module continues to select a number of data with the highest historical access frequency to migrate to the first storage cluster in descending order of historical access frequency.

[0054] The advantage of doing this is that data with high historical access frequency is more likely to be read and written multiple times by multiple clients in the future. Prioritizing the migration of data with high historical access frequency to the shared directory can utilize the high read and write speed of the first storage cluster to meet the needs of frequent reading and writing of such data, thereby improving data read and write efficiency.

[0055] like Figure 1 As shown, the method in this embodiment can be executed by the migration module of the first storage cluster. After the second storage cluster and the first storage cluster establish a communication connection in advance, the migration module can migrate the data to be shared in the second storage cluster to the shared directory of the first storage cluster for storage.

[0056] There can be one or more secondary storage clusters, with no limit on the number. Figure 1 For example, the migration module can obtain the data to be shared from the second storage cluster 1 and the second storage cluster 2. The data to be shared may include all or part of the data stored in the second storage cluster.

[0057] Given the capacity limitations of the first storage cluster, the first storage cluster can pre-allocate corresponding usable capacity for each second storage cluster in its own shared directory. Based on this, each time data to be shared is obtained, the corresponding amount of data to be shared can be obtained according to the usable capacity limit. New data to be shared can be obtained after there is spare usable capacity, until all the data of the second storage cluster is obtained.

[0058] In other words, the data to be shared stored in the second storage cluster can include:

[0059] Based on the available capacity pre-allocated in the shared directory for storing the data to be shared, the corresponding amount of data to be shared is read from the shared storage unit of the second storage cluster; wherein the amount of data to be shared matches the available capacity.

[0060] Combination Figure 1 For example, assuming the shared directory of the first storage cluster allocates 100 gigabytes (GB) of usable capacity to the second storage cluster 1, when the data to be shared is first retrieved, the migration module migrates 100GB of data from the second storage cluster 1 to the first storage cluster and stores it in the shared directory of the first storage cluster. At this point, the usable capacity of the second storage cluster 1 is full, and the migration module does not migrate data from the second storage cluster 1 to the first storage cluster immediately. Data migration from the second storage cluster 1 to the first storage cluster will continue when there is available usable capacity allocated to the first storage cluster.

[0061] The beneficial effect of this embodiment is that by allocating usable capacity to each second storage cluster, the amount of data migrated from the second storage cluster to the first storage cluster each time can be limited, thus avoiding excessive data migration to the first storage cluster and crowding out the storage resources of the first storage cluster.

[0062] When storing the migrated data to be shared in the first storage cluster, it can create several data blocks in its own storage resource pool, divide the data to be shared into several parts, and write them into these data blocks in the first storage cluster respectively.

[0063] The migration module can also migrate data stored in the shared directory back to the second storage cluster according to certain strategies, and delete this data from the shared directory of the first storage cluster after migration back to the second storage cluster. There are various strategies for migrating data from the shared directory back to the second storage cluster, and no specific strategy is required. For example, less frequently accessed data can be migrated back to the second storage cluster first.

[0064] When migrating data from a shared directory back to the second storage cluster, it is preferable to migrate it back to the second storage cluster to which it originally belonged. For example, if data read from the second storage cluster 1 is stored in the shared directory, and the first storage cluster wants to migrate this data back to the second storage cluster, then the data should be migrated back to the second storage cluster 1 first.

[0065] By executing the method of the above embodiments, when the first storage cluster receives a read request from a client, the first storage cluster can perform the following steps to provide data to the client:

[0066] In response to receiving a read request from a client, the system queries the shared directory, the first storage cluster, and the shared storage unit for the data to be read, based on the unified access point.

[0067] The data to be read is then sent back to the client.

[0068] If the data to be read comes from a shared storage unit, the data to be read is written to the shared directory to generate the corresponding shared data.

[0069] In this embodiment, if the data to be read comes from a shared directory, or from a directory other than the shared directory in the first storage cluster, then there is no need to perform the step of writing the data to be read to the shared directory; it is sufficient to simply send the data to be read back to the client.

[0070] The advantage of managing the shared directory in the above way is that data read by a client has a high probability of being read again by the same client or repeatedly read by other clients. By prioritizing the storage of data read by clients into the shared directory, the utilization rate of data in the shared directory can be improved, and the read and write performance of the first storage cluster can be fully utilized.

[0071] In some optional embodiments, if the second storage cluster processes client access requests (including at least one of read and write requests) based on a data sharing service before applying the method of this embodiment, then the method for obtaining the data to be shared stored in the second storage cluster can be:

[0072] Obtain the data to be shared stored in the second storage cluster from the original data sharing service of the second storage cluster;

[0073] Correspondingly, when obtaining data to be shared, the first storage cluster can also perform the following steps:

[0074] In the first storage cluster, a unified access entry (equivalent to a new data sharing service) is created based on the data to be shared. Access requests for the data to be shared are handled based on the unified access entry. The unified access entry and the original data sharing service contain the same storage directory.

[0075] The unified access entry point of the first storage cluster includes a storage directory that is equivalent to the aforementioned shared directory. The original data sharing service of the second storage cluster is equivalent to the shared storage unit of the second storage cluster in the previous embodiment.

[0076] The original data sharing service refers to the existing data sharing service for reading and writing data in the second storage cluster before executing the method of this embodiment. The new data sharing service refers to the data sharing service newly created by the first storage cluster for each second storage cluster, used to process access requests in the first storage cluster. The first storage cluster may include at least one new data sharing service, and the number of new data sharing services is the same as the number of second storage clusters and the two correspond one-to-one. For example, Figure 1 The second storage cluster 1 corresponds to the new data sharing service 1, and the second storage cluster 2 corresponds to the new data sharing service 2.

[0077] Data sharing services can be viewed as programs within a storage cluster used to read and write data from the storage cluster based on client access requests. These data sharing services can be based on Network Attached Storage (NAS) technology or other network sharing technologies.

[0078] Based on the creation of a new data sharing service, the method for migrating data to be shared from the second storage cluster to the first storage cluster can be as follows: Figure 3 As shown.

[0079] exist Figure 3 In the process, the migration module can first configure the source and target ends based on the operation and maintenance personnel's operation, that is, determine the network address of the source end and the network address of the target end. The source end can be the original data sharing service of the second storage cluster, and the target end can be the new data sharing service corresponding to the second storage cluster in the first storage cluster. For example, the source end is the original data sharing service of the second storage cluster 1, and the target end is the new data sharing service corresponding to the second storage cluster 1.

[0080] Once configured, the migration module begins the migration task. During the migration process, as long as there is available capacity in the first storage cluster, the migration module can read data from the source in batches and then write data to the target in batches. This process of batch reading and writing is repeated until the migration is complete, meaning until all the data that needs to be migrated from the source has been migrated to the shared directory of the target.

[0081] The method for creating a data sharing service can be found in the relevant technologies and will not be elaborated here. To ensure that the new data sharing service and the original data sharing service contain the same storage directory, when creating a new data sharing service, the storage directory of the original data sharing service can be exported and sent to the first storage cluster. This allows for the creation of a matching shared directory based on the exported storage directory, as well as a new data sharing service that reads and writes data based on that shared directory.

[0082] The above method for creating a shared directory is equivalent to creating a shared directory based on the data stored in the shared storage unit, wherein the shared directory matches the storage directory of the data in the shared storage unit.

[0083] The beneficial effect of this embodiment is that after migrating data from the second storage cluster to the first storage cluster according to the above method, since the shared directory of the new data sharing service is the same as the storage directory of the original data sharing service, the client can directly access the new data sharing service without updating the client's local storage directory. That is, the client does not need to perform additional business rectification operations, which improves the convenience of the method in this embodiment during implementation.

[0084] In the case of application data sharing services, a data sharing service can be created in each second storage cluster, for example... Figure 1 In the first storage cluster, Data Sharing Service 1 is created; in the second storage cluster, Data Sharing Service 2 is created. The Data Sharing Services provide an interface for the first storage cluster to access the second storage cluster, for example... Figure 1 On the one hand, it can receive data written by the first storage cluster through interfaces 1 and 2, write this data into the storage resource pool of the second storage cluster, and read and write the data stored in the storage resource pool of the second storage cluster according to the request sent by the first storage cluster from the interface.

[0085] In some optional embodiments, the first storage cluster can be set with a preset retention period. When the shared data from the second storage cluster stored in the shared directory has been continuously stored locally in the first storage cluster for the duration of the retention period, all of the data will be written to the corresponding second storage cluster.

[0086] Alternatively, the first storage cluster can also decide whether to migrate shared data back to the second storage cluster based on the frequency of use of the shared data. In other words, the first storage cluster can also be used for:

[0087] If the popularity of shared data is less than a preset popularity threshold, the shared data will be migrated back to the second storage cluster. The popularity of shared data represents the frequency with which the shared data is accessed.

[0088] Conversely, if the popularity of a piece of shared data is greater than or equal to the popularity threshold, the first storage cluster can continue to save this shared data in the local shared directory and will not move it back to the second storage cluster for the time being.

[0089] The above scheme is equivalent to the first storage cluster migrating the shared data from the shared directory to the shared storage unit of the second storage cluster when the popularity of the shared data is less than a preset popularity threshold. Popularity represents the frequency of access to the corresponding shared data.

[0090] The first storage cluster can periodically calculate the popularity of shared data and execute the above methods accordingly. The popularity of shared data can be represented by the cumulative number of times the shared data has been accessed within a preset time period up to the current moment. The preset time period can be set as needed, such as the last 24 hours, the last 30 hours, etc., without limitation. The specific value of the popularity threshold can also be set as needed, without limitation.

[0091] The beneficial effect of this embodiment is that after storing the shared data in the second storage cluster, reading the data from the second storage cluster will consume an additional amount of time. Writing the shared data with a popularity less than the popularity threshold to the second storage cluster can reduce the impact of the data reading time on the data reading performance, and ensure that the shared data with high popularity can be stored locally in the first storage cluster, thereby enabling the reading and writing of this shared data with high efficiency.

[0092] The data contained in the shared directory can be stored in various data blocks within the storage resource pool of the first storage cluster. The metadata database of the first storage cluster can record the mapping relationship between these data blocks and files, the logical storage address of these data blocks, and other metadata such as the last update time of the data stored in the data block, the data volume of the data stored in the data block, the compression type of the data stored in the data block, the garbage rate of the data stored in the data block, or other metadata.

[0093] by Figure 4 For example, the metadata database of the first storage cluster can be a key-value database. The metadata database can include a first data table and a second data table. The first data table, also denoted as the `Manifest_table`, records the mapping relationship between data blocks and files, such as which data blocks belong to file 1 and which belong to file 2. The second data table, also denoted as the `slog_table`, records the logical storage addresses of data blocks. Specifically, it can record information such as whether each data block is stored in a hot pool or a cold pool, if stored in a cold pool, which cold pool of the second storage cluster it is stored in, and its logical storage address within that cold pool.

[0094] Figure 4 In this context, "hot pool" refers to the storage resource pool of the first storage cluster, "cold pool" refers to the storage resource pool of the second storage cluster, and "cold pool list" is a status list maintained by the first storage cluster. The cold pool list can record the status and network address of the cold pools of each second storage cluster connected to the first storage cluster. The status of the cold pool can include whether the second storage cluster is operating normally, the maximum capacity and used capacity of the cold pool, and other possible status information.

[0095] like Figure 4 As shown, the first storage cluster also has a storage management module, which is used to migrate data from the shared directory back to the second storage cluster.

[0096] Optionally, the storage management module can be configured as follows: Figure 5 The method involves migrating cold data back to the cold pool of the second storage cluster. Cold data refers to data in the first storage cluster that belongs to the shared directory and / or does not belong to the shared directory, and whose popularity is less than the popularity threshold.

[0097] The first storage cluster can handle user write and read requests through front-end write and read service interfaces.

[0098] When a write request is received, the front-end write business interface executes B1, writes the data to the hot pool, and B2 updates the metadata. Specifically, when the front-end write business interface executes B1, it can write the data to be written carried by the write request to one or more data blocks in the hot pool of the first storage cluster. Then it can execute B2 to record the mapping relationship between these data blocks and files in the first data table and record the logical storage address of these data blocks in the second data table. At this point, the write request is processed.

[0099] Upon receiving a read request, the frontend read service interface queries the metadata based on the read request to determine if the read request is successful. If the metadata indicates that the data to be read is stored in the hot pool, the read request is considered successful; otherwise, it is considered unsuccessful. The data to be read refers to the data requested in the read request. In response to B3, the read request is successful, and the frontend read service interface can retrieve the data to be read and send it back to the client. At this point, the read request processing is complete.

[0100] The storage management module can respond to the front-end write business interface by writing new data and configuring the initial heat of the newly written data; it can respond to the front-end write business interface by performing write operations on the original data block and update the heat of the original data on which the write operation was performed; it can also respond to the front-end read business interface by executing B3 and correspondingly executing step B4 to update the heat of the data, that is, update the heat of the data read by the front-end read business interface.

[0101] As time goes by, the popularity of data in the first storage cluster will change, and some data may become cold data because they have not been accessed for a long time.

[0102] While maintaining the hotness of each data block in the hot pool in real time as described above, the storage management module periodically checks whether each data has become cold data. If it finds that data in the hot pool has become cold data, it responds to B5, detects cold data, and starts the migration process for these cold data to migrate them from the hot pool to the cold pool.

[0103] The migration process includes:

[0104] B6. Select a cold pool based on the remaining available capacity;

[0105] B7 migrates cold data to the cold pool;

[0106] B8, update the second data table to point to the cold pool.

[0107] When executing step B6, the storage management module can obtain the remaining available capacity of each cold pool from the key-value database, and select a cold pool based on the remaining available capacity. Specifically, it can select the cold pool with the largest remaining available capacity from multiple cold pools as the cold pool to be used for this migration (denoted as the target cold pool), or it can select the cold pool with the lowest utilization rate from multiple cold pools as the target cold pool. The utilization rate of a cold pool is equal to the remaining available capacity of the cold pool divided by the maximum capacity of the cold pool.

[0108] When executing B7, the storage management module can read data that has become cold data from the hot pool and write the read data into the selected target cold pool until all cold data in the hot pool has been written into the target cold pool and there is no cold data in the hot pool.

[0109] When executing B8, the storage management module can access the second data table stored in the key-value database, query the cold data written to the target cold pool in B7, change the location of the original cold data storage from the hot pool to the target cold pool, and change the logical storage address of the cold data from the original hot pool to the current logical storage address in the target cold pool. This completes the process of migrating the cold data from the first storage cluster back to the second storage cluster.

[0110] Through the above migration method, this embodiment can store frequently accessed hot data in the hot pool, utilize the high read and write speed of the first storage cluster to meet the needs of users to access hot data frequently, and migrate cold data that has not been accessed for a long time to the cold pool with low read and write speed. This can reduce the storage cost of this cold data and expand the amount of data stored.

[0111] It should be noted that step B6 is an optional step. In some embodiments, step B6 may not be performed. Instead, the cold data stored in the shared directory may be migrated back to the cold pool of the second storage cluster to which the cold data originally belonged.

[0112] The method for sending the data to be read back to the client and writing it to the shared directory in the foregoing embodiments can be found in [reference needed]. Figure 6 The front-end read service of the first storage cluster responds to the client's read request, executing step D1, whereby the front-end initiates a read request. Then, the front-end read service module executes D2 to read the mapping relationship of the first data table, and D3 to read the metadata of the data block. The front-end read service is implemented by the aforementioned unified access entry point, and is therefore part of that unified access entry point.

[0113] When executing D2 and D3, the front-end read module reads the first data table from the key-value database, and then reads the mapping relationships recorded in the first data table to determine which data blocks the data to be read indicated by the read request belongs to. These data blocks to which the data to be read belongs are the data blocks to be read. Specifically, the read request can carry the identifier of the data to be read, such as the inode corresponding to the data node to be read. The front-end read module can use this identifier to query the mapping relationship containing this identifier from the first data table, and query the mapping relationship to find out which data blocks the data is stored in. Then, it can read the metadata of these data blocks (such as the data block identifier slog_id) from the first data table.

[0114] Next, the front-end read module executes D4 to obtain the cold pool and logical storage address of the data block based on its metadata. During D4 execution, the front-end read module queries the mapping relationship recorded in the second data table based on the metadata of the data block to be read obtained in D3. This allows it to determine the cold pool where the data block to be read resides, and the logical storage address of the data block within the cold pool.

[0115] Finally, the front-end read module executes D5 based on the logical storage address, reads the data, and executes D6, then sends the data back to the front end. The front end is equivalent to the client mentioned above. When executing D5 and D6, the front-end read module calls the read interface of the cold pool where the data block to be read is located based on the logical storage address (this read interface is provided by the shared storage unit of the second storage cluster to which the cold pool belongs), thereby reading the data to be read stored in the data block to be read, and sending the read data back to the client that initiated the read request. At this point, the process of reading data from the shared storage unit of the second storage cluster and sending it back to the user ends.

[0116] Optionally, the data to be read from the second storage cluster can also be saved locally on the first storage cluster when it is fed back to the client, so that when the client reads this data again, it can be fed back directly from the shared directory of the first storage cluster, thereby improving the efficiency of feeding back data to the client.

[0117] Through the methods described in the above embodiments, the first storage cluster can feed back the cold data stored in the second storage cluster to the client according to the client's needs, thereby meeting the user's needs for persistent storage and querying cold data.

[0118] Figure 1 This embodiment provides a data storage device applied to a first storage cluster provided in any embodiment of this application. The first storage cluster includes a shared directory, and the data stored in the shared directory includes shared data from a second storage cluster, such as... Figure 7 As shown, the device includes:

[0119] Reading unit 701 reads the data to be shared from the shared storage unit of the second storage cluster based on the shared information of the second storage cluster.

[0120] The writing unit 702 is used to write the data to be shared to the shared directory and generate the corresponding shared data.

[0121] The specific working principle of the data storage device in this embodiment can be found in the relevant steps of the aforementioned data storage method, and will not be repeated here.

[0122] This application also provides an electronic device, including a memory and a processor;

[0123] Memory is used to store computer programs;

[0124] The processor is used to execute computer programs to implement the data storage method provided in any embodiment of this application.

[0125] This application also provides a computer storage medium for storing a computer program. When the computer program is executed by the processor of the electronic device to which the computer storage medium belongs, it is used to implement the data storage method provided in any embodiment of this application.

[0126] The working principles of the above-mentioned electronic devices and storage media can be found in the relevant steps of the data storage method in the foregoing embodiments, and will not be repeated here.

[0127] This embodiment also provides a data storage system, which includes a first storage cluster as provided in any embodiment of this application and at least one second storage cluster.

[0128] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A first storage cluster, characterized in that, The first storage cluster is used to connect to at least one second storage cluster; the storage rate of the first storage cluster is greater than the storage rate of the second storage cluster. The first storage cluster includes a shared directory, which is used to store at least the shared data of the second storage cluster; The first storage cluster provides a unified access point, which is used to access the shared data stored in the shared directory, the non-shared data of the first storage cluster, and the data stored in the shared storage units of the second storage cluster; the data stored in the shared storage units can be migrated to the shared directory.

2. The cluster according to claim 1, characterized in that, The first storage cluster is used for: Based on the shared information of the second storage cluster, read the data to be shared from the shared storage unit of the second storage cluster; The data to be shared is written to the shared directory to generate the corresponding shared data.

3. The cluster according to claim 1, characterized in that, The first storage cluster is also used for: If the popularity of the shared data is less than a preset popularity threshold, the shared data will be migrated from the shared directory to the shared storage unit of the second storage cluster. Popularity represents the frequency at which the corresponding shared data is accessed.

4. The cluster according to claim 2, characterized in that, The step of reading the data to be shared from the shared storage unit of the second storage cluster includes: Based on the available capacity pre-allocated in the shared directory for storing the data to be shared, the corresponding amount of data to be shared is read from the shared storage unit of the second storage cluster.

5. The cluster according to claim 2, characterized in that, The shared information includes the last update interval of the data in the shared storage unit of the second storage cluster; The step of reading the data to be shared from the shared storage unit of the second storage cluster based on the shared information of the second storage cluster includes: The data to be shared is read sequentially from the shared storage unit of the second storage cluster in ascending order of the last update interval, where the last update interval represents the time interval between the last time the corresponding data was updated and the current time.

6. The cluster according to claim 2, characterized in that, The shared information includes the historical access frequency of data in the shared storage units of the second storage cluster; The step of reading the data to be shared from the shared storage unit of the second storage cluster based on the shared information of the second storage cluster includes: The data to be shared is read sequentially from the shared storage units of the second storage cluster in descending order of historical access frequency.

7. The cluster according to claim 1, characterized in that, The first storage cluster is also used for: A shared directory is created based on the data stored in the shared storage unit, wherein the shared directory matches the storage directory of the data in the shared storage unit.

8. The cluster according to any one of claims 1 to 7, characterized in that, Also includes: In response to receiving a read request from a client, the system queries the shared directory, the first storage cluster, and the shared storage unit for the data to be read corresponding to the read request based on the unified access entry. The data to be read is then fed back to the client. If the data to be read comes from the shared storage unit, the data to be read is written to the shared directory to generate corresponding shared data.

9. A data storage device, characterized in that, Applied to a first storage cluster as described in any one of claims 1 to 8, the first storage cluster includes a shared directory, the data stored in the shared directory including shared data of a second storage cluster, the apparatus comprising: The reading unit reads the data to be shared from the shared storage unit of the second storage cluster based on the shared information of the second storage cluster; The writing unit is used to write the data to be shared into the shared directory to generate the corresponding shared data.

10. A data storage system, characterized in that, The storage system includes a first storage cluster as described in any one of claims 1 to 8 and at least one second storage cluster.

Images