Storage system expansion method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a capacity expansion method, device and equipment of a storage system and a storage medium. The method comprises the following steps: when a target device in a hot cluster included in the storage system needs to be expanded, scanning out a storage account from a local space, and keeping each historical business data of the storage account in the local space; determining metadata for describing a business data attribute of the storage account, and generating target routing information of the storage account; migrating the metadata to a data migration-in device corresponding to the target device, and storing the target routing information to an index cluster; after migrating the metadata and storing the target routing information to the index cluster, marking the storage account with a migration-out mark, so that the data migration-in device provides data read-write services of the storage account instead of the target device based on the metadata and the target routing information in the index cluster. The embodiments of the present application can effectively improve the expansion efficiency and ensure the stability of online services when the storage system is expanded.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for expanding a storage system. Background Technology

[0002] Currently, with the rapid growth in the amount of data that needs to be stored, storage systems are under significant pressure, with daily data transmission and reception exceeding tens of billions of records. However, in existing technologies, when any device in a storage system needs expansion, that device typically needs to transfer all its internal business data to a new device to achieve expansion of that device, and thus, the entire storage system. Because of the large volume of business data, this expansion method results in a long expansion cycle, and the process of migrating large amounts of business data consumes resources for online services, impacting their stability. Therefore, how to expand storage systems to improve expansion efficiency while ensuring the stability of online services has become a research hotspot. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for expanding a storage system, which can effectively improve expansion efficiency and ensure the stability of online services when expanding a storage system.

[0004] On one hand, embodiments of this application provide a method for expanding a storage system. The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster provides data read / write services for accounts, the cold cluster stores business data of accounts relocated from the hot cluster, and the index cluster stores routing information for the business data in the cold cluster. The method includes:

[0005] When a target device in the hot cluster needs to be expanded, the existing accounts are scanned from the local space of the target device, and the historical business data of each existing account are stored in the local space.

[0006] Determine the metadata used to describe the business data attributes of the existing account, and generate the target routing information of the existing account, wherein the target routing information is used to indicate the device where each historical business data of the existing account is located;

[0007] The metadata is migrated to the data migration device corresponding to the target device, and the target routing information is stored in the index cluster;

[0008] After migrating the metadata and storing the target routing information in the index cluster, the existing account is marked with a migration mark so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0009] On the other hand, this application provides another method for expanding a storage system. The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster provides data read / write services for accounts, the cold cluster stores business data of accounts residing in the hot cluster, and the index cluster stores routing information of the business data in the cold cluster. The method includes:

[0010] Receive metadata of existing accounts sent by the target device in the hot cluster, wherein the metadata is used to describe the business data attributes of the existing accounts;

[0011] The received metadata is stored in the local space, and after the target device marks the existing account with a migration mark, the data read and write service for the existing account is provided in place of the target device based on the metadata and the target routing information in the index cluster.

[0012] The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0013] On the other hand, embodiments of this application provide a storage system expansion device, the storage system including a hot cluster, a cold cluster, and an index cluster, the hot cluster being used to provide data read and write services for accounts, the cold cluster being used to store business data of accounts that have been migrated from the hot cluster, and the index cluster being used to store routing information of the business data in the cold cluster; the device includes;

[0014] The processing unit is used to scan the existing accounts from the local space of the target device when the target device in the hot cluster needs to be expanded, and to store the historical business data of the existing accounts in the local space.

[0015] The processing unit is further configured to determine metadata for describing the business data attributes of the existing account, and generate target routing information for the existing account, wherein the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0016] The output unit is used to migrate the metadata to the data migration device corresponding to the target device and store the target routing information in the index cluster;

[0017] The processing unit is further configured to mark the existing account with a migration mark after migrating the metadata and storing the target routing information in the index cluster, so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0018] On the other hand, this application embodiment provides another storage system expansion device, the storage system including a hot cluster, a cold cluster, and an index cluster, the hot cluster being used to provide data read and write services for accounts, the cold cluster being used to store business data of accounts relegated by the hot cluster, and the index cluster being used to store routing information of business data in the cold cluster; the device includes:

[0019] The receiving unit is used to receive metadata of existing accounts sent by the target device in the hot cluster, wherein the metadata is used to describe the business data attributes of the existing accounts;

[0020] The processing unit is used to store the received metadata in the local space, and after the target device marks the existing account with a migration mark, it provides data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0021] The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0022] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, wherein the memory is used to store computer programs;

[0023] In one embodiment, when the computer program is executed by the processor, it performs the following steps:

[0024] When a target device in the hot cluster needs to be expanded, the existing accounts are scanned from the local space of the target device, and the historical business data of each existing account are stored in the local space.

[0025] Determine the metadata used to describe the business data attributes of the existing account, and generate the target routing information of the existing account, wherein the target routing information is used to indicate the device where each historical business data of the existing account is located;

[0026] The metadata is migrated to the data migration device corresponding to the target device, and the target routing information is stored in the index cluster;

[0027] After migrating the metadata and storing the target routing information in the index cluster, the existing account is marked with a migration mark so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0028] In another embodiment, when the computer program is executed by the processor, it performs the following steps:

[0029] Receive metadata of existing accounts sent by the target device in the hot cluster, wherein the metadata is used to describe the business data attributes of the existing accounts;

[0030] The received metadata is stored in the local space, and after the target device marks the existing account with a migration mark, the data read and write service for the existing account is provided in place of the target device based on the metadata and the target routing information in the index cluster.

[0031] The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0032] On the other hand, embodiments of this application provide a computer storage medium storing a computer program;

[0033] In one embodiment, the computer program is adapted to be loaded by a processor and perform the following steps:

[0034] When a target device in the hot cluster needs to be expanded, the existing accounts are scanned from the local space of the target device, and the historical business data of each existing account are stored in the local space.

[0035] Determine the metadata used to describe the business data attributes of the existing account, and generate the target routing information of the existing account, wherein the target routing information is used to indicate the device where each historical business data of the existing account is located;

[0036] The metadata is migrated to the data migration device corresponding to the target device, and the target routing information is stored in the index cluster;

[0037] After migrating the metadata and storing the target routing information in the index cluster, the existing account is marked with a migration mark so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0038] In another embodiment, the computer program is adapted to be loaded by a processor and perform the following steps:

[0039] Receive metadata of existing accounts sent by the target device in the hot cluster, wherein the metadata is used to describe the business data attributes of the existing accounts;

[0040] The received metadata is stored in the local space, and after the target device marks the existing account with a migration mark, the data read and write service for the existing account is provided in place of the target device based on the metadata and the target routing information in the index cluster.

[0041] The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0042] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the aforementioned methods for expanding the storage system.

[0043] In this embodiment of the application, when a target device in a hot cluster of the storage system needs to be expanded, the target device can maintain the historical business data of each existing account in its local space. That is, it does not perform a migration operation on the existing historical business data. Instead, it migrates the metadata describing the number of business data of the storage account to the data migration-in device and stores the target routing information indicating the device where the historical business data of each existing account is located in the index cluster. This allows the data migration-in device to provide data read and write services for the existing account based on the metadata and the target routing information in the index cluster after the target device marks the existing account as migrated out, thereby completing the expansion of the target device. As can be seen, the embodiments of this application can achieve zero business data migration during the expansion process by reusing the cold and hot separation architecture of the storage system. By migrating a small amount of metadata and target routing information, the amount of data to be migrated can be effectively reduced, thereby improving expansion efficiency and enabling rapid expansion of the storage system to eliminate the bottleneck of hot cluster storage resources. Furthermore, since the entire expansion process does not require the migration of a large amount of business data, the problem of data migration competing for online service resources can be avoided. This ensures the stability of online services, enabling stable and rapid expansion even when the storage system capacity reaches the PB (a unit of storage capacity) level or the storage system is under high load. This will provide strong support for business growth and reduce the negative feedback rate of business. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1a This is a system architecture diagram of a storage system provided in an embodiment of this application;

[0046] Figure 1b This is a system architecture diagram of another storage system provided in an embodiment of this application;

[0047] Figure 1c This is a system architecture diagram of another storage system provided in the embodiments of this application;

[0048] Figure 1d This is a schematic diagram of a hash ring provided in an embodiment of this application;

[0049] Figure 1e This is a system architecture diagram of another storage system provided in the embodiments of this application;

[0050] Figure 2 This is a schematic flowchart of a storage system expansion method provided in an embodiment of this application;

[0051] Figure 3a This is a schematic diagram of business data provided in an embodiment of this application;

[0052] Figure 3b This is a schematic diagram of another type of business data provided in an embodiment of this application;

[0053] Figure 3c This is a system architecture diagram of another storage system provided in an embodiment of this application;

[0054] Figure 4 This is a flowchart illustrating another method for expanding the capacity of a storage system provided in an embodiment of this application;

[0055] Figure 5a This is a schematic diagram of a process for writing business data provided in an embodiment of this application;

[0056] Figure 5b This is a schematic diagram of another process for writing business data provided in an embodiment of this application;

[0057] Figure 5c This is a schematic diagram of target routing information provided in an embodiment of this application;

[0058] Figure 5d This is a schematic diagram of another target routing information provided in an embodiment of this application;

[0059] Figure 5e This is a schematic diagram of a process for reading business data provided in an embodiment of this application;

[0060] Figure 5f This is a schematic diagram illustrating the storage of target routing information provided in an embodiment of this application;

[0061] Figure 5g This is a schematic diagram of a process for accessing business data provided in an embodiment of this application;

[0062] Figure 5h This is a schematic diagram of a capacity expansion cycle provided in an embodiment of this application;

[0063] Figure 6 This is a schematic diagram of the structure of a storage system expansion device provided in an embodiment of this application;

[0064] Figure 7 This is a schematic diagram of the structure of another storage system expansion device provided in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0066] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0067] In this application embodiment, a data storage system based on a hot-cold separation architecture (hereinafter referred to as the storage system) is proposed; the storage system may include at least a hot cluster, a cold cluster, and an index cluster, etc. Specifically: ① A hot cluster refers to a device cluster used to store hot data of accounts and provide data read / write services for accounts. Hot data refers to frequently accessed business data (such as business data with an access frequency greater than a frequency threshold, or business data with an increasing access frequency). ② A cold cluster refers to a device cluster used to store cold data. Cold data refers to infrequently accessed business data (such as business data with an access frequency less than or equal to a frequency threshold, or business data with a decreasing access frequency). This infrequently accessed business data can be filtered from the stored business data by the hot cluster in real time or periodically and then sent to the cold cluster. Therefore, the cold cluster can be used to store the business data of accounts sent from the hot cluster. ③ An index cluster refers to a device cluster used to store routing information of business data in the cold cluster. Here, the routing information of business data refers to information used to indicate the device used to store the business data.

[0068] In one implementation, for any of the hot cluster, cold cluster, and index cluster, it may include one or more data synchronization groups. A data synchronization group refers to a group of devices (e.g., 3 or 6 devices) with data synchronization capabilities. In this case, the system architecture of the storage system can be found in [reference needed]. Figure 1a As shown. In this embodiment, Paxos (a message-passing-based consensus algorithm with high fault tolerance) can be used to achieve strong consistency data replication between devices in the same data synchronization group, thereby ensuring data consistency across devices within the same data synchronization group. In another implementation, for any of the hot cluster, cold cluster, and index cluster, it can directly include multiple devices, each storing different business data; in this case, the system architecture of the storage system can be found in [reference needed]. Figure 1b As shown.

[0069] It should be understood that, Figure 1a and Figure 1b This is merely an illustrative representation of the system architecture of the storage system and is not intended to limit it; and for ease of explanation, it will be referred to therein as... Figure 1a The storage system shown is used as an example for illustration. Furthermore, any device in the storage system can be a terminal device or a server, without limitation. Terminal devices mentioned here may include, but are not limited to: smartphones, tablets, laptops, desktop computers, smartwatches, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, etc.; various clients (apps) can run on these terminal devices, such as video playback clients, social media clients, browser clients, news feed clients, educational clients, etc.; servers mentioned here can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.; cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed.

[0070] See Figure 1c As shown, the storage system proposed in this application embodiment may involve write data streams and read data streams; wherein, the implementation principle of each data stream is as follows:

[0071] (1) The implementation principle of write data stream is as follows: When a terminal device on the object side needs to write business data of a certain account to the storage device, the terminal device on the object side finds the data synchronization group in the hot cluster used to store the business data of that account, and sends the business data to be written to one of the devices in the found data synchronization group. Then, the device receiving the business data can store the corresponding business data, and after storing the corresponding business data, it can synchronize the data with other devices in the same data synchronization group. Among them, the terminal device on the object side can access the hot cluster in the storage system through an internal client to send the business data to the corresponding device. In addition, the terminal device on the object side can find the corresponding data synchronization group in the hot cluster according to the correspondence between the account and the data synchronization group; or it can find the corresponding data synchronization group in the hot cluster through consistent hashing. There is no limitation on this.

[0072] It should be noted that the hash values ​​involved in consistent hashing fall within a preset range (e.g., 0-(2^32-1)), and this range can form a ring in space (i.e., a hash ring). The hash values ​​within this range can be distributed on the hash ring in either a clockwise or counter-clockwise direction; for ease of explanation, the following description will use a clockwise increasing direction. Based on this, each data synchronization group in the hot cluster can be distributed as a node on the hash ring. For any node on the hash ring, the data reading service for the account corresponding to the hash value (i.e., less than or equal to the hash value at the location of that node) located clockwise before that node is provided by the data synchronization group corresponding to that node. For details, see [link to documentation]. Figure 1d As shown, when a terminal device searches for a data synchronization group in a hot cluster that provides data read and write services for a certain account using consistent hashing, the terminal device can perform a hash calculation on the account identifier of the account to obtain the hash value corresponding to the account identifier, thereby obtaining the position of the hash value on the hash ring. Then, the terminal device can search for the node corresponding to the account (i.e., node C) in a clockwise direction to find the data synchronization group in the hot cluster that provides data read and write services for the account.

[0073] (2) The implementation principle of reading data stream is as follows: When a terminal device on the object side needs to read business data of a certain account from the storage device, the terminal device on the object side can find the data synchronization group in the hot cluster used to store the business data of that account, and send a read request for the business data to be read to one of the devices in the found data synchronization group; then the device receiving the read request can first search for the business data to be read in its local space. If it is found, it means that the business data to be read is hot data, and the found business data can be directly returned to the terminal device on the object side; if it is not found, it means that the business data to be read is cold data, and the routing information of the business data to be read is obtained from the index cluster, and the business data is read from the corresponding device in the cold cluster according to the obtained routing information. It can be seen that when the terminal device on the object side wants to access business data in the cold cluster, the device in the hot cluster needs to act as a proxy to read from the cold cluster according to the routing information stored in the index cluster.

[0074] Furthermore, the storage system proposed in this application embodiment can adopt a write-diffusion storage model, so that each account has a mailbox (i.e., a data sequence) for sending and receiving business data, each business data of any account has a data sequence number (seq), and the data sequence numbers of each business data of the same account are monotonically increasing, that is, the data sequence numbers of each business data of the same account are monotonically increasing within the corresponding mailbox. It should be understood that the hot cluster and cold cluster in the storage system can maintain independent data sequences for each account; then, during the process of the hot cluster providing data writing services for an account, the devices in the hot cluster can add newly added business data of any account to the end of the corresponding data sequence of the account, and assign a data sequence number to it based on the incrementing sequence number. Based on this, during the process of the hot cluster providing data reading services for an account, the terminal device can pull newly added business data according to the incremental data sequence number, or pull the corresponding business data according to the data sequence number range. Optionally, the storage system proposed in this application embodiment, in addition to data sinking service (that is, sinking business data in the hot cluster to the cold cluster), can also provide load balancing service and fault recovery service. Load balancing is a clustering technology that can distribute business data to multiple devices, thereby improving business processing capacity and ensuring high reliability of business. Fault recovery refers to restoring the stored data to the state before the fault occurred, and so on.

[0075] Based on the above description, this application proposes a method for expanding a storage system. The general principle of this method is as follows: When a target device in a hot cluster needs to be expanded, the target device can be regarded as an old machine or a data migration-out device (or migration-out machine), and a new device is introduced into the storage system as the data migration-in device (or migration-in machine) corresponding to the target device. Figure 1eAs shown; then, the hot cluster can be treated as part of the cold cluster, so that the historical business data before the expansion is still retained in the target device, and the corresponding target routing information is added to the index cluster, so that the data migration device can determine the device where the historical business data is located based on the routing information in the index cluster, and the metadata used to describe the business data attributes of the corresponding account is migrated to the data migration device. Then, the data migration device can finally provide data read and write services for the corresponding account on behalf of the target device based on the metadata and the target routing information in the index cluster. In this way, the expansion is equivalent to a special sinking, which not only retains the simplicity and stability of static routing, but also reuses the cold and hot separation architecture of the storage system.

[0076] The following is combined with Figure 2 The flowchart shown further illustrates the storage system expansion method proposed in the embodiments of this application; specifically, as... Figure 2 As shown, the method for expanding the storage system may include the following steps S201-S206:

[0077] S201 When a target device in a hot cluster needs to be expanded, the target device scans out the existing accounts from its local space and stores the historical business data of each existing account in its local space.

[0078] Here, "existing accounts" refers to accounts associated with at least one set of historical business data in the local space of the target device. "Historical business data" refers to the business data stored in the local space of the target device during expansion. For existing accounts, the target device can provide a proactive migration service. This proactive migration service involves actively scanning the local space to obtain a list of unmigrated existing accounts, migrating the metadata of each existing account in the list, and without relocating the historical business data of the corresponding existing accounts. Therefore, when executing step S201, the target device can use the proactive migration service to scan for existing accounts in its local space and maintain the storage of each historical business data item for each existing account in the local space.

[0079] It is worth noting that since the writing of new business data does not depend on historical business data, and the business data cannot be modified after it is written, the embodiments of this application can, during capacity expansion, allow the target device to segment the historical business data of existing accounts according to the data sequence number, so that the historical business data of existing accounts can be retained on the target device.

[0080] It should be noted that the storage system expansion method proposed in this application is applicable to various application scenarios, such as social scenarios, file service scenarios, and music service scenarios, etc. For example, when the application scenario is a social scenario, the account mentioned in the embodiments of this application can be a social account, and the business data stored in the storage system refers to: messages generated by each object through the corresponding social account during the social process, such as... Figure 3a As shown; for example, when the application scenario is a file service scenario, the account mentioned in the embodiments of this application can be a file management account, and the business data stored in the storage system refers to: the files generated or obtained by each object through the corresponding file management account, such as Figure 3b As shown. For example, in a music service scenario, the account mentioned in this application embodiment can be a music account, and the business data stored in the storage system refers to music files created or downloaded by each object through its corresponding music account, etc. This application does not limit the type of account or the type of business data.

[0081] S202, the target device determines the metadata used to describe the business data attributes of the existing account and generates the target routing information for the existing account.

[0082] The target routing information is used to indicate the device where each historical service data of the existing account is located. Specifically, the target routing information may include the communication address (i.e., machine location) of the device where each historical service data of the existing account is located; the communication address here can be, for example, an IP address (Internet Protocol Address) or a MAC address (Media Access Control) address, etc.

[0083] In this application embodiment, the business data attributes described by the metadata include, but are not limited to: the amount of unread business data, the amount of received business data, the amount of sent business data, and the total amount of business data, etc.; this application does not limit the specific content included in the business data attributes described by the metadata. It should be noted that, since the metadata is used to describe the business data attributes of existing accounts, that is, the metadata includes a small number of fields such as business data counting results, the data volume of the metadata of existing accounts can be much smaller than the data volume of the historical business data of existing accounts. This is so that the amount of data to be migrated can be reduced by migrating the metadata of existing accounts without migrating the historical business data of existing accounts.

[0084] In one specific implementation, the target device can generate metadata for the existing account when writing business data to it for the first time, and store this metadata in its local space. Furthermore, during the process of providing data read / write services for the existing account, if an update requirement for the existing account's metadata is detected, the target device can update the metadata in its local space. In this case, when the target device determines the metadata used to describe the business data attributes of the existing account through step S202, it can directly obtain the metadata from its local space.

[0085] In another specific implementation, when the target device first writes business data to an existing account, it may not perform a metadata generation operation. Instead, metadata is generated in real time when there is a definite need for it. In this case, when the target device determines the metadata used to describe the attributes of the business data of the existing account in step S202, it can generate the metadata of the existing account based on the data information of the business data of the existing account stored in the local space and the data information of the business data of the existing account stored in the cold cluster. The data information of the business data of the stored existing account includes at least one of the following: the quantity of business data of the stored existing account, the amount of business data sent and received by the stored existing account, and the data status of each business data of the stored existing account (such as read status, unread status), etc. It should be noted that when the target device determines the metadata through step S202, the business data of the existing accounts stored in the local space of the target device shall at least include the various historical business data mentioned in step S201; optionally, it may also include the new business data of the existing accounts, which is received by the target device after executing step S201 and before executing step S202.

[0086] S203, the target device will migrate the metadata of the existing account to the corresponding data migration device.

[0087] Specifically, the target device can send the metadata of the existing account to the corresponding data migration device. In turn, after receiving the metadata of the existing account sent by the target device, the data migration device can store the received metadata in its local space.

[0088] S204, the target device stores the target routing information to the index cluster.

[0089] Specifically, the target device can send target routing information to the index cluster. Correspondingly, after receiving the target routing information from the target device, the index cluster can store the received target routing information. Since the target routing information includes the communication addresses of the devices where the historical service data of each existing account is located, the index cluster can store the communication addresses of the devices where the historical service data of the existing account is stored by storing the target routing information.

[0090] It should be noted that during the process of storing target routing information in the index cluster, if the index cluster already stores target routing information for existing accounts, and in this case, the target routing information already stored in the index cluster is used to indicate the devices where each cold service data (i.e., service data stored in the cold cluster) of the existing account is located, then the index cluster can use the received target routing information to update the stored target routing information, obtaining the updated target routing information. The updated target routing information is used to indicate the devices where each historical service data and each cold service data of the existing account is located. In other words, after the index cluster stores the received target routing information, the target routing information in the index cluster can be used to indicate the devices where each historical service data and each cold service data of the existing account is located. If the index cluster does not store target routing information for existing accounts, then the index cluster can store the received target routing information, so that the target routing information in the index cluster can be used to indicate the devices where each historical service data of the existing account is located.

[0091] S205: After the target device migrates metadata and stores target routing information in the index cluster, the target device marks the existing accounts with a migration mark.

[0092] It should be noted that when the target device marks an existing account as a move-out account, the target device can record the move-out mark in the metadata of the existing account to mark the existing account as a move-out account; or, the target device can generate a mark list, and the mark list is used to indicate that any account stored in the mark list is marked as a move-out account. Then the target device can add existing accounts to the mark list to mark the existing accounts as a move-out account, and so on; this application does not limit this.

[0093] S206, the data migration device provides data read and write services for existing accounts on behalf of the target device, based on metadata and target routing information in the index cluster.

[0094] It should be understood that, in this embodiment of the application, when the data migration device provides data read / write services for existing accounts on behalf of the target device based on metadata and target routing information in the index cluster, if an update requirement for the metadata of the existing accounts is detected, the data migration device can update the metadata of the existing accounts so that it can subsequently provide data read / write services for existing accounts based on the updated metadata. The data read / write services for existing accounts include: data reading services and data writing services. The data reading service mentioned herein may further include at least one of the following: a service for reading business data of existing accounts, and a service for reading the unread volume, received volume, sent volume, and total quantity of business data of existing accounts; the data writing service mentioned herein may further include at least one of the following: a service for writing business data of existing accounts.

[0095] When the data reading service includes reading business data from existing accounts, since the business data of existing accounts may include new business data (i.e., business data stored on the data migration device), historical business data, and cold business data (i.e., business data stored in a cold cluster), when providing data reading services for new business data of existing accounts, the data migration device can read the corresponding business data in its local space and return the read business data to the terminal device. Similarly, when providing data reading services for historical and cold business data of existing accounts, the data migration device can, based on the target routing information in the index cluster, proxy-read the corresponding business data in the target device or cold cluster and return it to the terminal device. Figure 3c As shown in the figure. Among them, data block routing refers to recording routing information in the form of data blocks; it should be noted that the specific implementation process of the data migration device replacing the target device to provide data reading services for existing accounts is as shown in the following embodiments, which will not be repeated here.

[0096] In the embodiments of this application, Figure 3c This illustration merely demonstrates the provision of data reading services for existing accounts by the data migration device, and this application does not limit the scope of the illustration. For example, both the target device and the data migration device can be independent servers. Alternatively, the data migration device can store target routing information indicating the location of each historical business data item for the existing account. In this case, when providing data reading services for the historical business data of the existing account, the data migration device can act as an agent to read the corresponding business data from the target device based on the target routing information in its local space, and so on.

[0097] Correspondingly, when the data migration device replaces the target device to provide data writing services for existing accounts, the data migration device can receive data addition requests sent by the terminal device. These data addition requests are used to request the writing of incremental business data for existing accounts. The data migration device can then respond to the data addition requests and write the aforementioned incremental business data into the local space of the data migration device. In other words, the data migration device can store incremental business data in its local space.

[0098] Furthermore, during the process of providing data writing services for existing accounts by the data migration device, if the business data attribute described by the metadata of the existing account includes the total quantity of business data, then the data migration device can update the total quantity described by the metadata of the existing account after writing the incremental business data; or, if the business data attribute described by the metadata of the existing account includes the amount of business data received, and the aforementioned incremental business data is business data received by the existing account, then the data migration device can update the amount of data received described by the metadata of the existing account after writing the incremental business data; or, if the business data attribute described by the metadata of the existing account includes the amount of business data not yet read, and the aforementioned incremental business data is unread business data, then the data migration device can update the amount of unread data described by the metadata of the existing account after writing the incremental business data, and so on. It should be understood that after writing the aforementioned incremental business data, if the data migration device detects a change in at least one business data attribute described by the metadata, that is, detects a need to update the metadata of the existing account, then it can update the corresponding business data attribute described by the metadata of the existing account.

[0099] In this embodiment of the application, when a target device in a hot cluster of the storage system needs to be expanded, the target device can maintain the historical business data of each existing account in its local space. That is, it does not perform a migration operation on the existing historical business data. Instead, it migrates the metadata describing the number of business data of the storage account to the data migration-in device and stores the target routing information indicating the device where the historical business data of each existing account is located in the index cluster. This allows the data migration-in device to provide data read and write services for the existing account based on the metadata and the target routing information in the index cluster after the target device marks the existing account as migrated out, thereby completing the expansion of the target device. As can be seen, the embodiments of this application can achieve zero business data migration during the expansion process by reusing the cold and hot separation architecture of the storage system. By migrating a small amount of metadata and target routing information, the amount of data to be migrated can be effectively reduced, thereby improving expansion efficiency and enabling rapid expansion of the storage system to eliminate the bottleneck of hot cluster storage resources. Furthermore, since the entire expansion process does not require the migration of a large amount of business data, the problem of data migration competing for online service resources can be avoided. This ensures the stability of online services, enabling stable and rapid expansion even when the storage system capacity reaches the PB (a unit of storage capacity) level or the storage system is under high load. This will provide strong support for business growth and reduce the negative feedback rate of business.

[0100] Based on the above description, this application also proposes a more specific method for expanding a storage system. This storage system may include a hot cluster, a cold cluster, and an index cluster. The hot cluster provides data read / write services for accounts, the cold cluster stores business data of accounts relocated from the hot cluster, and the index cluster stores routing information for the business data in the cold cluster. Please refer to [link to previous document]. Figure 4 The method for expanding the storage system may include the following steps S401-S411:

[0101] S401 When a target device in a hot cluster needs to be expanded, the target device scans the existing accounts from its local space and stores the historical business data of each existing account in its local space.

[0102] It should be noted that, in addition to storing the historical business data of existing accounts, the local space of the target device can also receive data write requests during the expansion process to store new business data of existing accounts or new business data of incremental accounts, etc.; among which, incremental accounts refer to accounts that are not associated with historical business data.

[0103] Based on this, during the expansion process, the target device can receive a data write request sent by the terminal device. This data write request requests the writing of business data for a first account into the target device, and the data write request can carry the business data to be written to the target device. Based on this, the target device can respond to the data write request and detect the account type of the first account based on the accounts already stored in its local space. An account type includes an incremental account type or a stock account type. The incremental account type indicates that the corresponding account is an incremental account, and the stock account type indicates that the corresponding account is a stock account. Accordingly, if the first account's account type is an incremental account type, the target device can mark the first account as migrated out and perform the data write operation for the first account based on the business data carried in the data write request; if the first account's account type is a stock account type, the target device can perform the data write operation for the first account based on the business data carried in the data write request.

[0104] In the specific implementation, if the first account is an incremental account, the target device can also determine whether the business data carried in the data write request includes the first business data of the first account. If so, the step of marking the first account with a migration mark is triggered. It is evident that an incremental account will trigger passive migration when business data is written for the first time. In this case, the target device will mark the incremental account with a migration mark and perform the corresponding data write operation.

[0105] Optionally, the aforementioned data write request may also carry the data sequence number of the corresponding business data and the account identifier of the first account, or it may also carry the account identifier of the first account; then the target device can detect the account type of the first account based on the account identifier of the first account and the account identifiers of accounts already stored in the local space. Here, the account identifier of any account can refer to any account or to the account ID (identity number) of any account; this application does not limit this.

[0106] Specifically, when detecting the account type of the first account based on the account identifier of the first account and the account identifiers of accounts already stored in the local space, if the account identifiers of accounts already stored in the local space of the target device do not include the account identifier of the first account, or if the account identifiers of accounts already stored in the local space include the account identifier of the first account and all business data associated with the first account are new business data, that is, none of the business data associated with the first account are historical business data, then the target device can determine that the account type of the first account is an incremental account type; if the account identifiers of accounts already stored in the local space of the target device include the account identifier of the first account and the first account is associated with at least one historical business data, then the target device can determine that the account type of the first account is an existing account type.

[0107] Furthermore, when the target device performs a data write operation for the first account based on the service data carried in the data write request, the target device can write the service data carried in the data write request to its local space and determine the communication address of the data migration device. Based on this, the target device can return the communication address of the data migration device to the terminal device, so that the terminal device can write the service data carried in the data write request to the data migration device based on the communication address of the data migration device, such as... Figure 5a As shown in the diagram, during the expansion process (i.e., migration), new business data is written to both the target device and the data migration-in device to support rollback functionality. In this case, the new business data is first written to the target device and then to the data migration-in device. After the expansion is complete, the terminal device can directly write the new business data to the data migration-in device. It should be noted that, without considering dual writing, the target device only provides data read services during the expansion process, allowing the data migration-in device to take over the subsequent data write services for new business data.

[0108] Accordingly, when a data write request carries the account identifier of the first account and the data sequence number of the carried business data, the target device can perform a data write operation for the first account based on the business data carried in the data write request, the account identifier of the first account, and the data sequence number of the carried business data. Specifically, the target device can write the business data carried in the data write request into its local space according to the account identifier of the first account and the data sequence number of the carried business data. That is, the target device can determine the data sequence of the first account according to the account identifier of the first account and write the data sequence number of the carried business data and the business data carried in the data write request into the data sequence of the first account in its local space. Alternatively, when a data write request carries the account identifier of the first account, the target device can determine the data sequence of the first account based on the account identifier of the first account, assign a data sequence number to the business data carried in the data write request, and then write the carried business data and the corresponding data sequence number into the data sequence of the first account in its local space.

[0109] Similarly, the data migration device can also write the business data carried in the data write request to the local space based on the account identifier of the first account carried in the data write request and the data sequence number of the business data carried in the data write request; or, the data migration device can also write the business data carried in the data write request to the local space based on the account identifier of the first account carried in the data write request.

[0110] For example, such as Figure 5b As shown, assuming the data write request carries not only the business data but also the account identifier of the first account, when the target device receives the data write request from the terminal device, it can determine the data sequence of the first account according to the account identifier carried in the data write request, and assign data sequence number 13 to the business data carried in the data write request, thereby writing the business data carried in the data write request at the 13th position of the data sequence of the first account. Correspondingly, the terminal device can send a data write request to the data migration device based on the received communication address of the data migration device, so that the data migration device can write the business data carried in the data write request into its local space based on the received account identifier. It should be noted that... Figure 5b This application only illustrates the dual-write operation exemplarily and does not limit its scope. For example, in addition to the business data carried in the data write request, the data sequence number of the business data carried in the data write request may also be carried. For another example, during the write operation on the first account, the target device may also return the data sequence number of the business data carried in the data write request to the terminal device, and so on.

[0111] S402, the target device determines the metadata of the existing account and generates the target routing information for the existing account.

[0112] It should be noted that each historical business data of an existing account has a unique data sequence number, and the data sequence numbers of each historical business data of an existing account are different from each other.

[0113] In one implementation, when generating target routing information for an existing account, the target device can determine the sequence number range composed of the data sequence numbers of each historical service data of the existing account; and generate the target routing information for the existing account based on the account identifier, the sequence number range, and the communication address of the target device. Accordingly, the target routing information may include the account identifier, the sequence number range, and the communication address of the target device.

[0114] For example, assuming the existing account's account identifier is account 1, the target device's communication address is 100.100.111, and the data sequence numbers of each historical service data of the existing account range from 1 to 6, then the sequence number range formed by the data sequence numbers of each historical service data of the existing account is 1-6; in this case, the target routing information of the existing account can be as follows: Figure 5c As shown.

[0115] In another implementation, when generating target routing information for an existing account, the target device can determine the data sequence number of each historical service data of the existing account, and generate the target routing information for the existing account based on the account identifier of the existing account, the data sequence number of each historical service data of the existing account, and the communication address of the target device. In this case, the target routing information includes the account identifier of the existing account, the data sequence number of each historical service data of the existing account, and the communication address of the target device.

[0116] For example, suppose the account identifier of the existing account is account 1, the communication address of the target device is 100.100.111, and the data sequence numbers of each historical service data of the existing account include 1-6; in this case, the target routing information of the existing account can be as follows: Figure 5d As shown.

[0117] It should be noted that, Figure 5c and Figure 5d The target routing information is only illustrated by example and is not limited thereto. For example, the account identifier may be located after the data sequence number in the target routing information. Or, when the target routing information includes routing information for each historical business data of an existing account, the target routing information may include only the account identifier of an existing account, and this account identifier corresponds to the data sequence number of each historical business data, and so on.

[0118] S403, the target device migrates its metadata to the corresponding data migration device.

[0119] In other words, the data import device can receive metadata sent by the target device and store the received metadata in its local space.

[0120] It should be noted that the metadata of existing accounts includes a version number, and this version number monotonically increments as the metadata changes. Based on this, after receiving the metadata from the target device, the data migration device can search for the metadata of existing accounts in its local space. If the corresponding metadata is found in the local space, the data migration device determines the version number in the found metadata as the first version number and the version number in the received metadata as the second version number. If the second version number is greater than the first version number, the step of storing the received metadata in the local space is triggered. In this case, the data migration device can use the received metadata to update the found metadata. Conversely, if the corresponding metadata is not found in the local space, the data migration device can trigger the step of storing the received metadata in the local space.

[0121] In this embodiment of the application, the method of processing the received metadata based on the version number in the metadata can be called a version mechanism, which can be used to prevent old metadata from overwriting new metadata.

[0122] S404, the target device stores the target routing information in the index cluster.

[0123] Specifically, after the index cluster receives the target routing information sent by the target device, the index cluster can store the target routing information. If the index cluster already stores the target routing information of existing accounts, it can use the received target routing information to update the stored target routing information, thereby realizing the storage of the received target routing information.

[0124] It should be understood that if the target routing information received by the index cluster may include the account identifier of the existing account, the sequence number range composed of the data sequence numbers of each historical business data of the existing account, and the communication address of the target device, and the stored target routing information may include the account identifier of the existing account, the sequence number range composed of the data sequence numbers of each cold business data of the existing account, and the communication address of the device in the cold cluster where the cold business data of the existing account is located, then after storing the received target routing information, the target routing information in the index cluster may include the account identifier of the existing account, the sequence number range composed of the data sequence numbers of each historical business data of the existing account, the communication address of the target device, the sequence number range composed of the data sequence numbers of each cold business data of the existing account, and the communication address of the device in the cold cluster where the cold business data of the existing account is located.

[0125] Correspondingly, if the target routing information received by the index cluster may include the account identifier of the existing account, the data sequence number of each historical business data of the existing account, and the communication address of the target device, and the stored target routing information may include the account identifier of the existing account, the data sequence number of each cold business data of the existing account, and the communication address of the device in the cold cluster where the cold business data of the existing account is located, then after storing the received target routing information, the target routing information in the index cluster may include the account identifier of the existing account, the data sequence number of each historical business data of the existing account, the communication address of the target device, the data sequence number of each cold business data of the existing account, and the communication address of the device in the cold cluster where the cold business data of the existing account is located.

[0126] S405: After the target device migrates metadata and stores target routing information in the index cluster, the target device marks existing accounts with a migration mark.

[0127] It should be noted that during the expansion process, the terminal device can send a data read request to the target device. That is, the target device can receive the data read request sent by the terminal device, which requests to read the target business data of the second account. Then, the target device can respond to the data read request by detecting the migration mark of the second account in its local space and obtaining the mark detection result. Based on this, the target device can generate response information corresponding to the data read request according to the mark detection result and return the response information to the terminal device. Furthermore, the terminal device can obtain the target business data based on the response information. It should be understood that if marking any account as a migration mark means recording the migration mark in the metadata of that account, then the target device can detect the migration mark of the second account in the metadata of the second account included in its local space. Moreover, when the terminal device requests to read the target business data based on the account identifier of the second account and the data sequence number of the target business data, the aforementioned data read request can carry the account identifier of the second account and the data sequence number of the target business data.

[0128] In specific implementation, when the target device generates the response information corresponding to the data read request based on the tag detection result, if the tag detection result indicates that a migration tag for the second account has been detected, the target device generates the response information corresponding to the data read request using the communication address of the data migration-in device. In this case, the response information may include the communication address of the data migration-in device. Then, the terminal device can send a data read request to the data migration-in device based on the communication address of the data migration-in device to obtain the target service data from the data migration-in device, such as... Figure 5eAs shown, during the expansion process, when a terminal device reads the account's business data, it can first route to the target device. If the target device detects that the corresponding account has been marked as migrated out, it can redirect the terminal device to the data migration-in device to obtain the corresponding business data.

[0129] Correspondingly, if the tag detection result indicates that no migration tag for the second account was detected, the target device obtains the target business data from the local space and uses the target business data to generate the response information corresponding to the data read request; in this case, the response information may include the target business data, and the terminal device can obtain the target business data from the response information.

[0130] In this embodiment of the application, the target device can also obtain access information of each historical business data of the existing account. Any access information includes at least one of the following: the storage duration of the corresponding historical business data in the local space, and the target reading frequency of the corresponding historical business data within the target time period. Based on this, the target device can select the historical business data to be sinked from each historical business data of the existing account according to the access information of each historical business data obtained. The selected historical business data is then sinked to the cold cluster.

[0131] It should be noted that the storage system may also include a sinking service, which is deployed on the same machine as the hot cluster. In order to free up space for writing new business data, the target device can select the historical business data to be sinked through the sinking service, and sink the selected historical business data as cold business data to the cold cluster. Therefore, in this embodiment, after a period of time, local space on the target device can be freed up, thereby enabling the hot cluster to have sufficient space to store new business data.

[0132] Furthermore, the target device can also determine the communication address of the device where the selected historical service data is stored in the cold cluster, and update the target routing information in the index cluster according to the determined communication address; or, after receiving the selected historical service data, the device receiving the sinking historical service data can determine the corresponding communication address to update the index cluster, etc.; this application does not limit this.

[0133] In specific implementations, the target device selects the historical business data to be migrated from the historical business data of the existing account based on the access information of the acquired historical business data. The implementation methods include, but are not limited to, the following:

[0134] In the first implementation, when any access information includes a target read frequency, the target device can obtain the historical read frequency of the m-th historical business data within a historical time period. The end time of this historical time period is earlier than or equal to the start time of the target time period, where m is a positive integer and not greater than the total number of historical business data in the existing account. Then, the target device can predict the direction of change of the read frequency of the m-th historical business data based on the historical read frequency and the target read frequency in the access information of the m-th historical business data. If the predicted direction of change is a frequency decrease, then the m-th historical business data is selected as the historical business data to be relegated. In this case, after a certain period of time, the read frequency (i.e., access frequency) of the historical business data decreases, so the target device can determine the historical business data with the decreasing frequency as the historical business data to be relegated.

[0135] Specifically, when the historical reading frequency is less than the target reading frequency, the target device can determine that the predicted direction of change is a frequency decrease; or, when the difference between the historical reading frequency and the target reading frequency is greater than a preset frequency threshold, the target device can determine that the predicted direction of change is a frequency decrease, and so on; this application does not limit this. The preset frequency threshold can be set based on experience or based on actual needs; this application does not limit this.

[0136] It should be understood that the duration indicated by the aforementioned historical time period may be the same as or different from the duration indicated by the target time period, and this application does not limit this; furthermore, this application does not limit the specific time indicated by the historical time period and the target time period. For example, the target time period may be: the first 5 minutes when the target device obtains the access information of the corresponding historical business data, or it may be: the first 10 minutes when the target device obtains the access information of the corresponding historical business data, and so on.

[0137] In the second implementation, when any access information includes storage duration, the target device can determine the storage duration of the m-th historical service data. If the storage duration of the m-th historical service data exceeds a preset duration threshold, the target device can select the m-th historical service data as the historical service data to be migrated. In this case, the target device can periodically migrate the historical service data to the cold cluster through the migration service.

[0138] The preset duration threshold can be set based on experience or actual needs; this application does not limit this setting. For example, the preset duration threshold can be one hour, one day, etc.

[0139] In the third implementation, when any access information includes storage duration and target read frequency, the target device can obtain the historical read frequency of the m-th historical service data within a historical time period, and determine the target read frequency of the m-th historical service data within a target time period, as well as the storage duration of the m-th historical service data. Based on this, the target device can predict the direction of change of the read frequency of the m-th historical service data according to the historical read frequency and the target read frequency. If the predicted direction of change is a frequency decrease direction, and the storage duration of the m-th historical service data is greater than a preset duration threshold, the target device can select the m-th historical service data as the historical service data to be pushed down, and so on.

[0140] Furthermore, after the selected historical business data is pushed down, the target device can delete the selected historical business data; specifically, the target device can increase the data sequence number of the pushed down position. Based on this, when the target device stores data based on an LSM tree (Log-Structured Merge Tree), as the LSM tree data is compressed, data with a data sequence number lower than the data sequence number of the pushed down position will be deleted by the target device.

[0141] It should be noted that the aforementioned hot cluster may include one or more data synchronization groups, and the devices within the same data synchronization group maintain data consistency; and when any data synchronization group needs to be expanded, all devices in the corresponding data synchronization group are expanded simultaneously. Since expansion requires modification of three independent systems—the target device, the index cluster, and the data migration device—and the data synchronization groups for these three systems can use Paxos to achieve strong consistency data replication, concurrency or data loss in any system may lead to incomplete data, thus failing to guarantee the consistency of metadata between multiple systems; therefore, this application embodiment avoids concurrent migration by allowing a single account to be initiated serially by a single machine (i.e., a single device).

[0142] Based on this, the target device can belong to one of the devices in the target data synchronization group that needs to be expanded in the hot cluster; the migration of a single existing account in the target data synchronization group is executed serially by a single device in the target data synchronization group. The migration of a single existing account includes: migrating the metadata of the corresponding existing account, storing the target routing information of the corresponding existing account in the index cluster, and marking the corresponding existing account as migrated out.

[0143] It should be understood that since the migration of a single account can be divided into multiple steps, it is impossible to guarantee that each step will be successful. Therefore, this application embodiment may initiate the migration of the same account repeatedly, and a special design has been made in the migration process. It will only be perceived by the business when the last step (i.e., marking the migration out) is completed. In other words, the migration work will only be determined to be completed when the existing account is marked with the migration out mark, so that the interruption of any step will not affect the correctness of the final result.

[0144] S406 After the target device marks the existing account as migrated out, the terminal device can send a data access request to the data migration-in device.

[0145] In other words, the data migration device can receive data access requests sent by terminal devices; these data access requests are used to request the reading of specified business data from existing accounts. It should be noted that the data access request can carry the account identifier of the existing account and the data sequence number of the specified business data.

[0146] It is worth noting that after the expansion is completed, the terminal device can directly find the data migration device through the routing information. In this case, when the terminal device requests to read the specified business data, the data migration device in the hot cluster can first try to read the specified business data in the local space. When the specified business data to be read is stored in the target device or cold cluster, the data migration device will read the routing information of the specified business data from the index cluster, and then request the target device or cold cluster to return the specified business data. The specific implementation process is as follows.

[0147] S407, the data migration device responds to the data access request and searches for the specified business data in the local space.

[0148] Specifically, the data migration device can search for the specified business data in the local space according to the account identifier of the existing account and the data sequence number of the specified business data; in this case, the data migration device can search for the data sequence of the existing account in the local space according to the account identifier of the existing account, and search for the specified business data from the data sequence of the existing account according to the data sequence number of the specified business data.

[0149] It should be understood that if the data sequence of the existing account includes the data sequence number of the specified business data, the data migration device can find the specified business data; if the data sequence of the existing account does not include the data sequence number of the specified business data, the data migration device cannot find the specified business data.

[0150] S408 If the data migration device cannot find the specified business data, the data migration device requests the index cluster to return the routing information of the specified business data based on the stored target routing information.

[0151] Specifically, if the data migration device cannot find the specified business data, it can send a request to the index cluster to obtain the routing information of the specified business data. After receiving the request from the data migration device, the index cluster can determine the routing information of the specified business data and return the routing information of the specified business data to the data migration device.

[0152] The aforementioned request can carry the account identifier of the existing account and the data sequence number of the specified business data. The index cluster can then determine the routing information of the specified business data based on the account identifier of the existing account and the data sequence number of the specified business data.

[0153] S409, the data migration device obtains the specified business data from the corresponding device based on the routing information returned by the index cluster.

[0154] The device indicated by the routing information returned by the index cluster (i.e., the routing information of the specified business data) can be the target device or a device in the cold cluster; this application does not limit this.

[0155] Based on this, the data migration device can send the data access request to the corresponding device (i.e., the device indicated by the routing information returned by the index cluster) based on the routing information returned by the index cluster; accordingly, after receiving the data access request, the device can obtain the specified business data in the local space and return the specified business data to the data migration device.

[0156] In this embodiment, when business data is migrated to the local storage location, the index cluster is used to store routing information for cold business data. Cold business data is typically from a week ago and has a very low access frequency (i.e., read frequency), resulting in low access pressure on the index cluster. However, during rapid expansion, the index cluster can be used to store target routing information for existing accounts generated by the target device. At this time, the historical business data of the existing accounts in the target device has not yet become "cold" (i.e., it is not considered cold business data). In this case, the access frequency of the historical business data of the existing accounts is high, creating access hotspots for the index cluster and leading to high access pressure. Therefore, the data migration device can store the target routing information of existing accounts in its local space to reduce the access pressure on the index cluster, thereby eliminating tens of millions of index accesses per minute.

[0157] Specifically, the data migration device can obtain the target routing information of existing accounts from the target device or index cluster; and after obtaining the target routing information, it stores the target routing information in local space. Optionally, the data migration device can store the target routing information in the metadata of the existing accounts in local space, that is, add a previous-hop index cache to the metadata of the existing accounts in local space to record the sequence number range composed of the data sequence numbers of each historical business data of the existing account, as well as the communication address of the target device, such as... Figure 5f As shown.

[0158] It should be understood that, Figure 5f This illustration merely demonstrates how the data migration device stores target routing information, and this application does not limit this; for example, the data migration device may store the target routing information in a local space outside the storage space where the metadata is located; or, the data migration device may store the routing information of each historical business data separately, and so on.

[0159] It should be noted that historical business data that does not need to be migrated during the expansion process needs to be moved to the index cluster after it becomes cold (i.e., after being treated as cold business data) (usually after a week). After being moved, the routing information of the corresponding business data will change, which will cause the target routing information stored in the local space of the data migration device to become invalid. Therefore, it is necessary to pull the corresponding routing information again from the index cluster.

[0160] In one implementation, if the specified service data is not found, the data migration device can determine the routing information for the specified service data based on the target routing information stored in its local space, and request the corresponding device (i.e., the target device) to return the specified service data based on the determined routing information. If an error indicator code is received from the corresponding device, the step of requesting the index cluster to return the routing information for the specified service data is triggered. The error indicator code indicates that the specified service data has been moved to a cold cluster. Upon receiving the error indicator code, the data migration device can retrieve the routing information from the index cluster again based on the error indicator code.

[0161] In another implementation, the target routing information in the local space is configured with a validity period. If the specified service data is not found, the data migration device can count the storage time of the target routing information in the local space. Based on the storage time and the validity period, the device performs a validity check on the target routing information in the local space and obtains a check result. If the check result indicates that the target routing information in the local space has expired (i.e., the storage time is longer than the validity period), the step of requesting the index cluster to return the routing information of the specified service data is triggered. The validity period can be set based on experience or actual needs; this application does not limit this. The validity period can also be referred to as Time To Live (TTL).

[0162] Correspondingly, if the detection result indicates that the target routing information in the local space of the data migration device is not invalid, the data migration device can request the corresponding device to return the specified business data based on the routing information obtained from the local space; furthermore, if an error indicator code is received from the corresponding device, the step of requesting the index cluster to return the routing information of the specified business data is triggered.

[0163] It should be understood that if the data migration device obtains the routing information of the specified service data from the local space, and reads the specified service data from the corresponding device based on the obtained routing information, then the data migration device may not execute steps S408 and S409.

[0164] S410, the data transfer device returns the acquired specified business data to the terminal device.

[0165] It should be noted that if the data migration device finds the specified business data, the data migration device can return the specified business data to the terminal device. In other words, the data migration device can return the specified business data stored in the local space to the terminal device. In this case, steps S408 and S409 may not be executed in this embodiment of the application.

[0166] The flowchart illustrating the process by which a terminal device sends a data access request to a data migration device to obtain specified business data from an existing account is as follows: Figure 5g As shown. It should be noted that, Figure 5gThis application merely illustrates a schematic diagram of the process by which a terminal device accesses specified business data from a data migration device, and does not limit the scope of the application. For example, when the data migration device finds the specified business data in its local space, it can return the found specified business data to the terminal device without obtaining the routing information of the specified business data. Or, when the target routing information is stored in the local space of the data migration device, the data migration device can obtain the routing information of the specified business data in its local space without obtaining the routing information of the specified business data from the index cluster, and so on.

[0167] It should be noted that the business data attributes described by the metadata of existing accounts may include: the amount of unread business data; based on this, if the data migration device detects a need to update the metadata of existing accounts, the data migration device can update the metadata of existing accounts, as shown in the following specific implementation method.

[0168] S411, after the data migration device returns the specified service data to the terminal device, if the specified service data is being read for the first time, the data migration device updates the unread amount described in the metadata.

[0169] It should be understood that since the business data attributes described by the metadata of the existing account include the unread amount of business data, the specified business data is unread before it is read for the first time. Correspondingly, when the specified business data is read for the first time, the specified business data is changed from unread business data to read business data. Then the data migration device can detect the update requirement of the metadata of the existing account and update the unread amount described by the metadata of the existing account.

[0170] It should be understood that if the business data attributes described by the metadata of an existing account include the read count of business data, then after the data migration device returns the specified business data to the terminal device, if the specified business data is being read for the first time, the data migration device can update the read count described by the metadata; or, if the business data attributes described by the metadata of an existing account include the read count and unread count of business data, then after the data migration device returns the specified business data to the terminal device, if the specified business data is being read for the first time, the data migration device can update the read count and unread count described by the metadata, and so on.

[0171] Furthermore, to better illustrate the beneficial effects of the storage system expansion method proposed in this application, this application also conducts experiments using a KV (Key-Value) data migration and expansion method as an example. Compared to expanding the total number of devices storing the same historical business data, the expansion efficiency of the storage system expansion method proposed in this application is improved by tens of times, enabling the storage system to achieve expansion capabilities at the minute level. This reduces the expansion cycle from one week to minutes. Figure 5h As shown. Furthermore, this expansion, through the storage system expansion method proposed in this application, rapidly migrated hundreds of millions of existing accounts to the data migration device. During the expansion process, there was no fluctuation in upstream and downstream services, meaning that the normal use of the objects was not affected, resulting in zero complaints and maintaining availability above five nines (i.e., 99.999%).

[0172] This application embodiment allows for the following expansion mechanism: when a target device in a hot cluster needs to be expanded, the target device can maintain the storage of historical business data for existing accounts in its local space. That is, it does not perform migration operations on the stored historical business data. Instead, it migrates the metadata of the existing accounts to the data migration-in device and stores the target routing information of the existing accounts in the index cluster. This enables the data migration-in device to provide data read / write services for the existing accounts on behalf of the target device, based on the metadata and the target routing information in the index cluster, after marking the existing accounts as migrated out. Therefore, this application embodiment can shorten the expansion cycle by migrating only the metadata and target routing information, without migrating or sinking business data, thus achieving rapid expansion and effectively improving expansion efficiency. Furthermore, after the data migration device receives a data access request from the terminal device requesting to read specified business data of an existing account, if the data migration device cannot find the specified business data in its local space but has target routing information stored in its local space, the data migration device can obtain the routing information of the specified business data in its local space to read the specified business data from the corresponding device, which can effectively reduce the access pressure on the index cluster. Correspondingly, if the target routing information stored in the local space of the data migration device becomes invalid, the data migration device can re-retrieve the routing information of the specified business data from the index cluster to obtain the specified business data from the device where the specified business data is located. In addition, during the expansion process, this application embodiment can achieve multi-system consistency through a method of serially initiating migration of a single account from a single device, a version mechanism, and a repeatable migration method. Furthermore, during the expansion process, this application embodiment can store the newly added business data in both the target device and the data migration device simultaneously through a dual-write method, enabling this application embodiment to support a rollback function.

[0173] Based on the description of the relevant embodiments of the storage system expansion method above, this application also proposes a storage system expansion device. The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster provides data read / write services for accounts, the cold cluster stores business data of accounts residing in the hot cluster, and the index cluster stores routing information for the business data in the cold cluster. The storage system expansion device can be a computer program (including program code) running on a computer device (i.e., the target device). Please refer to... Figure 6 The expansion device of this storage system can operate the following units:

[0174] The processing unit 601 is used to scan the existing accounts from the local space of the target device when the target device in the hot cluster needs to be expanded, and to store the historical business data of the existing accounts in the local space.

[0175] The processing unit 601 is further configured to determine metadata for describing the business data attributes of the existing account, and generate target routing information for the existing account, wherein the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0176] Output unit 602 is used to migrate the metadata to the data migration device corresponding to the target device and store the target routing information in the index cluster;

[0177] The processing unit 601 is further configured to mark the existing account with a migration mark after migrating the metadata and storing the target routing information in the index cluster, so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0178] In one implementation, each historical service data of the existing account has a data sequence number, and the data sequence numbers of each historical service data of the existing account are different from each other; when generating the target routing information of the existing account, the processing unit 601 may specifically be used to:

[0179] Determine the range of sequence numbers composed of the data sequence numbers of each historical business data of the existing account;

[0180] Based on the account identifier of the existing account, the sequence number range, and the communication address of the target device, the target routing information of the existing account is generated.

[0181] In another embodiment, the expansion device of the storage system further includes an input unit 603; the input unit 603 can be used for:

[0182] During the expansion process, a data write request is received from a terminal device. The data write request is used to request the writing of the business data of the first account into the target device.

[0183] Processing unit 601 can also be used for:

[0184] In response to the data write request, the account type of the first account is detected based on the accounts already stored in the local space;

[0185] If the first account is an incremental account, then mark the first account as moved out, and perform the data writing operation for the first account based on the business data carried by the data writing request.

[0186] If the first account is an existing account, then the data writing operation for the first account is performed based on the business data carried in the data writing request.

[0187] In another embodiment, when the processing unit 601 performs the data writing operation for the first account based on the business data carried by the data writing request, it may specifically be used to:

[0188] Write the business data carried by the data write request into the local space, and determine the communication address of the data migration device;

[0189] The communication address of the data migration device is returned to the terminal device, so that the terminal device can write the service data carried in the data writing request to the data migration device based on the communication address of the data migration device.

[0190] In another embodiment, the input unit 603 can also be used for:

[0191] During the expansion process, a data read request is sent by the receiving terminal device. The data read request is used to request the reading of the target business data of the second account.

[0192] Processing unit 601 can also be used for:

[0193] In response to the data read request, the migration marker of the second account is detected in the local space, and the marker detection result is obtained;

[0194] Generate response information corresponding to the data reading request based on the marker detection results;

[0195] Output unit 602 can also be used for:

[0196] The response information is returned to the terminal device so that the terminal device can obtain the target service data based on the response information.

[0197] In another embodiment, when the processing unit 601 generates the response information corresponding to the data reading request based on the marker detection result, it may specifically be used to:

[0198] If the marker detection result indicates that a migration marker for the second account has been detected, then the communication address of the data migration device is used to generate the response information corresponding to the data read request;

[0199] If the marker detection result indicates that no migration marker for the second account was detected, then the target business data is obtained from the local space, and the target business data is used to generate the response information corresponding to the data read request.

[0200] In another embodiment, the processing unit 601 may also be used for:

[0201] Obtain access information for each historical business data of the existing account, wherein any access information includes at least one of the following: the storage duration of the corresponding historical business data in the local space, and the target reading frequency of the corresponding historical business data within the target time period;

[0202] Based on the access information of each historical business data obtained, select the historical business data that needs to be pushed down from each historical business data of the existing account.

[0203] The selected historical business data will be migrated to the cold cluster.

[0204] In another implementation, any access information includes a target read frequency; when the processing unit 601 selects the historical business data to be pushed down from the historical business data of the existing account based on the access information of each historical business data obtained, it can be specifically used for:

[0205] Obtain the historical read frequency of the m-th historical business data within a historical time period, wherein the end time of the historical time period is earlier than or equal to the start time of the target time period, and m is a positive integer and not greater than the total number of historical business data of the existing account.

[0206] Based on the historical read frequency and the target read frequency in the access information of the m-th historical service data, predict the direction of change of the read frequency of the m-th historical service data;

[0207] If the predicted direction of change is a decrease in frequency, then the m-th historical service data is selected as the historical service data to be downgraded.

[0208] In another implementation, the hot cluster includes one or more data synchronization groups, and the devices in the same data synchronization group maintain data consistency; and when any data synchronization group needs to be expanded, the devices in the corresponding data synchronization group are expanded simultaneously.

[0209] The target device belongs to one of the target data synchronization groups that need to be expanded in the hot cluster; the migration of a single existing account in the target data synchronization group is performed serially by a single device in the target data synchronization group.

[0210] The migration of a single existing account includes: migrating the metadata of the corresponding existing account, storing the target routing information of the corresponding existing account in the index cluster, and marking the corresponding existing account as migrated out.

[0211] According to one embodiment of this application, when the computer device is the target device... Figure 2 or Figure 4 The steps involving the target device as the executing entity in the method shown can all be performed by... Figure 6 This is performed by each unit in the expansion device of the storage system shown. For example, Figure 2 Steps S201, S202, and S205 shown can all be generated by... Figure 6 The processing unit 601 shown executes the steps S203 and S204, which can both be performed by... Figure 6 The output unit 602 shown is executed. For example, Figure 4 Steps S401, S402, and S405 shown can all be generated by... Figure 6 The processing unit 601 shown executes the steps S403 and S404, which can both be performed by [the processor / organization]. Figure 6 The output unit 602 shown executes, etc.

[0212] According to another embodiment of this application, Figure 6 The various units in the storage system expansion device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the storage system expansion device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0213] According to another embodiment of this application, when the target device is a computer device, it is possible to execute functions such as those described above by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Figure 2 or Figure 4 The corresponding methods shown involve computer programs (including program code) for each execution subject being the target device, to construct, as... Figure 6 The diagram illustrates a storage system expansion device and a storage system expansion method for implementing embodiments of this application. The computer program may be recorded on, for example, a computer storage medium, loaded onto the aforementioned computing device via the computer storage medium, and run therein.

[0214] In this embodiment of the application, when a target device in a hot cluster of the storage system needs to be expanded, the target device can maintain the historical business data of each existing account in its local space. That is, it does not perform a migration operation on the existing historical business data. Instead, it migrates the metadata describing the number of business data of the storage account to the data migration-in device and stores the target routing information indicating the device where the historical business data of each existing account is located in the index cluster. This allows the data migration-in device to provide data read and write services for the existing account based on the metadata and the target routing information in the index cluster after the target device marks the existing account as migrated out, thereby completing the expansion of the target device. As can be seen, the embodiments of this application can achieve zero business data migration during the expansion process by reusing the cold and hot separation architecture of the storage system. By migrating a small amount of metadata and target routing information, the amount of data to be migrated can be effectively reduced, thereby improving expansion efficiency and enabling rapid expansion of the storage system to eliminate the bottleneck of hot cluster storage resources. Furthermore, since the entire expansion process does not require the migration of a large amount of business data, the problem of data migration competing for online service resources can be avoided. This ensures the stability of online services, enabling stable and rapid expansion even when the storage system capacity reaches the PB (a unit of storage capacity) level or the storage system is under high load. This will provide strong support for business growth and reduce the negative feedback rate of business.

[0215] Based on the above-described storage system expansion method, this application also proposes another storage system expansion device; wherein the storage system includes a hot cluster, a cold cluster, and an index cluster, the hot cluster is used to provide data read and write services for accounts, the cold cluster is used to store business data of accounts residing in the hot cluster, and the index cluster is used to store routing information of the business data in the cold cluster; the storage system expansion device can be a computer program (including program code) running on a computer device (i.e., a data migration device). Please see... Figure 7The expansion device of this storage system can operate the following units:

[0216] The receiving unit 701 is used to receive metadata of existing accounts sent by the target device in the hot cluster, wherein the metadata is used to describe the business data attributes of the existing accounts;

[0217] The processing unit 702 is used to store the received metadata in the local space, and after the target device marks the existing account with a migration mark, it provides data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

[0218] The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located.

[0219] In one implementation, the metadata describes the business data attributes of the existing account, including: the amount of unread business data;

[0220] When processing unit 702 provides data read / write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster, it can be specifically used for:

[0221] Receive a data access request sent by a terminal device, the data access request being used to request the reading of specified business data of the existing account;

[0222] In response to the data access request, locate the specified business data in the local space;

[0223] If the specified service data is found, the specified service data is returned to the terminal device;

[0224] If the specified service data is not found, the index cluster is requested to return the routing information of the specified service data according to the stored target routing information, and the specified service data is obtained from the corresponding device based on the routing information returned by the index cluster, and the obtained specified service data is returned to the terminal device.

[0225] After returning the specified service data to the terminal device, if the specified service data is being read for the first time, the unread amount described in the metadata is updated.

[0226] In another embodiment, the processing unit 702 may also be used for:

[0227] Obtain the target routing information of the existing account from the target device or the index cluster;

[0228] After obtaining the target routing information, the target routing information is stored in the local space.

[0229] In another implementation, if the specified service data is not found, the processing unit 702 can also be used for:

[0230] Based on the target routing information stored in the local space, determine the routing information of the specified service data, and request the corresponding device to return the specified service data based on the determined routing information;

[0231] If an error code is received from the corresponding device, the step of requesting the index cluster to return the routing information of the specified business data is triggered.

[0232] In another implementation, the target routing information in the local space is configured with a valid duration; if the specified service data is not found, the processing unit 702 can also be used for:

[0233] Calculate the duration for which the target routing information has been stored in the local space;

[0234] Based on the stored duration and the valid duration, the target routing information in the local space is validated to obtain the validation result.

[0235] If the detection result indicates that the target routing information in the local space has expired, then the step of requesting the index cluster to return the routing information of the specified business data is triggered.

[0236] In another implementation, the metadata of existing accounts includes a version number, and the corresponding version number monotonically increments when the metadata changes; the processing unit 702 can also be used for:

[0237] After receiving the metadata sent by the target device, search for the metadata of the existing account in the local space;

[0238] If the corresponding metadata is found in the local space, the version number in the found metadata is determined as the first version number, and the version number in the received metadata is determined as the second version number.

[0239] If the second version number is greater than the first version number, then the step of storing the received metadata in the local space is triggered.

[0240] According to one embodiment of this application, when the computer device is a data transfer device... Figure 2 or Figure 4 The steps involving data migration devices as the executing entities in the method shown can all be performed by... Figure 7This is performed by each unit in the expansion device of the storage system shown. For example, Figure 2 Step S206 shown can be performed by Figure 7 The processing unit 702 shown executes this. For example, Figure 4 Steps S407-S411 shown can all be performed by Figure 7 The processing unit 702 shown executes, etc.

[0241] According to another embodiment of this application, Figure 7 The various units in the storage system expansion device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the storage system expansion device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0242] According to another embodiment of this application, when the target device is a computer device, it is possible to execute functions such as those described above by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Figure 2 or Figure 4 The corresponding methods shown involve computer programs (including program code) for steps performed by data transfer devices, to construct... Figure 7 The diagram illustrates a storage system expansion device and a storage system expansion method for implementing embodiments of this application. The computer program may be recorded on, for example, a computer storage medium, loaded onto the aforementioned computing device via the computer storage medium, and run therein.

[0243] In this embodiment of the application, when a target device in a hot cluster of the storage system needs to be expanded, the target device can maintain the historical business data of each existing account in its local space. That is, it does not perform a migration operation on the existing historical business data. Instead, it migrates the metadata describing the number of business data of the storage account to the data migration-in device and stores the target routing information indicating the device where the historical business data of each existing account is located in the index cluster. This allows the data migration-in device to provide data read and write services for the existing account based on the metadata and the target routing information in the index cluster after the target device marks the existing account as migrated out, thereby completing the expansion of the target device. As can be seen, the embodiments of this application can achieve zero business data migration during the expansion process by reusing the cold and hot separation architecture of the storage system. By migrating a small amount of metadata and target routing information, the amount of data to be migrated can be effectively reduced, thereby improving expansion efficiency and enabling rapid expansion of the storage system to eliminate the bottleneck of hot cluster storage resources. Furthermore, since the entire expansion process does not require the migration of a large amount of business data, the problem of data migration competing for online service resources can be avoided. This ensures the stability of online services, enabling stable and rapid expansion even when the storage system capacity reaches the PB (a unit of storage capacity) level or the storage system is under high load. This will provide strong support for business growth and reduce the negative feedback rate of business.

[0244] Based on the description of the above method and apparatus embodiments, this application also provides a computer device. Please refer to... Figure 8 The computer device includes at least a processor 801, an input interface 802, an output interface 803, and a computer storage medium 804. The processor 801, input interface 802, output interface 803, and computer storage medium 804 within the computer device can be connected via a bus or other means.

[0245] The computer storage medium 804 can be stored in the memory of the computer device. The computer storage medium 804 is used to store computer programs, which include program instructions. The processor 801 is used to execute the program instructions stored in the computer storage medium 804. The processor 801 (or CPU (Central Processing Unit)) is the computing and control core of the computer device. It is adapted to implement one or more instructions, specifically to load and execute one or more instructions to achieve corresponding method flows or corresponding functions.

[0246] In one embodiment, when the computer device is the target device, the processor 801 described in this application embodiment can be used to perform a series of storage system expansions, specifically including: when the target device in the hot cluster needs to be expanded, scanning out existing accounts from the local space of the target device, and storing each historical business data of the existing account in the local space; determining metadata used to describe the business data attributes of the existing account, and generating target routing information for the existing account, the target routing information being used to indicate the device where each historical business data of the existing account is located; migrating the metadata to the data migration device corresponding to the target device, and storing the target routing information in the index cluster; after migrating the metadata and storing the target routing information in the index cluster, marking the existing account with a migration-out mark, so that the data migration device, based on the metadata and the target routing information in the index cluster, provides data read and write services for the existing account on behalf of the target device, etc.

[0247] In another embodiment, when the computer device is a data migration device, the processor 801 described in this application embodiment can be used to perform a series of storage system expansions, specifically including: receiving metadata of existing accounts sent by the target device in the hot cluster, the metadata being used to describe the business data attributes of the existing accounts; storing the received metadata in local space, and after the target device marks the existing accounts with a migration-out mark, providing data read and write services for the existing accounts on behalf of the target device based on the metadata and the target routing information in the index cluster; wherein, the target routing information is stored by the target device in the index cluster, and the target routing information is used to indicate the devices where the various historical business data of the existing accounts are located, etc.

[0248] This application embodiment also provides a computer storage medium (memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer storage medium provides storage space that stores the operating system of the computer device. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor. In one embodiment, when the computer device is the target device, the processor can load and execute one or more instructions stored in the computer storage medium to achieve the above-mentioned... Figure 2 or Figure 4 In one embodiment of the storage system expansion method, the execution entities are the target devices and the method steps; in another embodiment, when the computer device is a data import device, the processor can load and execute one or more instructions stored in the computer storage medium to achieve the above-mentioned... Figure 2 or Figure 4 In the embodiments of the storage system expansion method shown, each execution entity is a method step of the data migration device.

[0249] In this embodiment of the application, when a target device in a hot cluster of the storage system needs to be expanded, the target device can maintain the historical business data of each existing account in its local space. That is, it does not perform a migration operation on the existing historical business data. Instead, it migrates the metadata describing the number of business data of the storage account to the data migration-in device and stores the target routing information indicating the device where the historical business data of each existing account is located in the index cluster. This allows the data migration-in device to provide data read and write services for the existing account based on the metadata and the target routing information in the index cluster after the target device marks the existing account as migrated out, thereby completing the expansion of the target device. As can be seen, the embodiments of this application can achieve zero business data migration during the expansion process by reusing the cold and hot separation architecture of the storage system. By migrating a small amount of metadata and target routing information, the amount of data to be migrated can be effectively reduced, thereby improving expansion efficiency and enabling rapid expansion of the storage system to eliminate the bottleneck of hot cluster storage resources. Furthermore, since the entire expansion process does not require the migration of a large amount of business data, the problem of data migration competing for online service resources can be avoided. This ensures the stability of online services, enabling stable and rapid expansion even when the storage system capacity reaches the PB (a unit of storage capacity) level or the storage system is under high load. This will provide strong support for business growth and reduce the negative feedback rate of business.

[0250] It should be noted that, according to one aspect of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer storage medium. When a computer device is the target device, the processor of the computer device reads the computer instructions from the computer storage medium, executes the computer instructions, and causes the computer device to perform the aforementioned actions. Figure 2 or Figure 4 The storage system expansion method embodiments shown are provided in various optional ways for the target device to perform the execution; when the computer device is a data import device, the processor of the computer device reads the computer instructions from the computer storage medium, the processor executes the computer instructions, causing the computer device to perform the above-described... Figure 2 or Figure 4 The various execution subjects of the storage system expansion method embodiments shown are methods provided in the optional mode of data migration into the device.

[0251] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A method for expanding the capacity of a storage system, characterized in that, The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster provides data read / write services for accounts, the cold cluster stores business data of accounts originating from the hot cluster, and the index cluster stores routing information for the business data in the cold cluster. The method includes: When a target device in the hot cluster needs to be expanded, existing accounts are scanned from the local space of the target device, and historical business data of each existing account are stored in the local space; wherein, after the target device is expanded, at least one historical business data of the existing account is moved to the cold cluster. Determine metadata for describing the business data attributes of the existing account, and generate target routing information for the existing account. The target routing information is used to indicate the device where each historical business data of the existing account is located, and the device includes the target device. The metadata is migrated to the data migration device corresponding to the target device, and the target routing information is stored in the index cluster; After migrating the metadata and storing the target routing information in the index cluster, the existing account is marked with a migration mark so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

2. The method according to claim 1, characterized in that, Each historical business data point of the existing account has a unique data sequence number; the generation of target routing information for the existing account includes: Determine the range of sequence numbers composed of the data sequence numbers of each historical business data of the existing account; Based on the account identifier of the existing account, the sequence number range, and the communication address of the target device, the target routing information of the existing account is generated.

3. The method according to claim 1, characterized in that, The method further includes: During the expansion process, a data write request is received from a terminal device. The data write request is used to request the writing of the business data of the first account into the target device. In response to the data write request, the account type of the first account is detected based on the accounts already stored in the local space; If the first account is an incremental account, then mark the first account as moved out, and perform the data writing operation for the first account based on the business data carried by the data writing request. If the first account is an existing account, then the data writing operation for the first account is performed based on the business data carried in the data writing request.

4. The method according to claim 3, characterized in that, The step of performing the data writing operation for the first account based on the business data carried in the data writing request includes: Write the business data carried by the data write request into the local space, and determine the communication address of the data migration device; The communication address of the data migration device is returned to the terminal device, so that the terminal device can write the service data carried in the data writing request to the data migration device based on the communication address of the data migration device.

5. The method according to claim 1, characterized in that, The method further includes: During the expansion process, a data read request is sent by the receiving terminal device. The data read request is used to request the reading of the target business data of the second account. In response to the data read request, the migration marker of the second account is detected in the local space, and the marker detection result is obtained; Based on the marker detection result, a response information corresponding to the data reading request is generated, and the response information is returned to the terminal device so that the terminal device can obtain the target service data based on the response information.

6. The method according to claim 5, characterized in that, The step of generating response information corresponding to the data reading request based on the marker detection result includes: If the marker detection result indicates that a migration marker for the second account has been detected, then the communication address of the data migration device is used to generate the response information corresponding to the data read request; If the marker detection result indicates that no migration marker for the second account was detected, then the target business data is obtained from the local space, and the target business data is used to generate the response information corresponding to the data read request.

7. The method according to claim 1, characterized in that, The method further includes: Obtain access information for each historical business data of the existing account, wherein any access information includes at least one of the following: the storage duration of the corresponding historical business data in the local space, and the target reading frequency of the corresponding historical business data within the target time period; Based on the access information of each historical business data obtained, select the historical business data that needs to be pushed down from each historical business data of the existing account. The selected historical business data will be migrated to the cold cluster.

8. The method according to claim 7, characterized in that, Any access information includes the target read frequency; the step of selecting the historical business data to be pushed down from the historical business data of the existing account based on the access information of each historical business data obtained includes: Obtain the historical read frequency of the m-th historical business data within a historical time period, wherein the end time of the historical time period is earlier than or equal to the start time of the target time period, and m is a positive integer and not greater than the total number of historical business data of the existing account. Based on the historical read frequency and the target read frequency in the access information of the m-th historical service data, predict the direction of change of the read frequency of the m-th historical service data; If the predicted direction of change is a decrease in frequency, then the m-th historical service data is selected as the historical service data to be downgraded.

9. The method according to claim 1, characterized in that, The hot cluster includes one or more data synchronization groups, and the devices in the same data synchronization group maintain data consistency; and when any data synchronization group needs to be expanded, the devices in the corresponding data synchronization group are expanded simultaneously. The target device belongs to one of the target data synchronization groups that need to be expanded in the hot cluster; the migration of a single existing account in the target data synchronization group is performed serially by a single device in the target data synchronization group. The migration of a single existing account includes: migrating the metadata of the corresponding existing account, storing the target routing information of the corresponding existing account in the index cluster, and marking the corresponding existing account as migrated out.

10. A method for expanding the capacity of a storage system, characterized in that, The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster is used to provide data read and write services for accounts. The cold cluster is used to store business data of accounts that are carried over from the hot cluster. The index cluster is used to store routing information of business data in the cold cluster. The method includes: The system receives metadata of existing accounts sent by the target device in the hot cluster. The metadata describes the business data attributes of the existing accounts. The existing accounts are scanned from the local space of the target device when the target device needs to be expanded. Each historical business data of the existing account is stored in the local space when the target device needs to be expanded. After the target device is expanded, at least one historical business data of the existing account is moved to the cold cluster. The received metadata is stored in the local space, and after the target device marks the existing account with a migration mark, the data read and write service for the existing account is provided in place of the target device based on the metadata and the target routing information in the index cluster. The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located, including the target device.

11. The method according to claim 10, characterized in that, The metadata describes the business data attributes of the existing accounts, including: the amount of unread business data; The provision of data read / write services for the existing accounts based on the metadata and target routing information in the index cluster, replacing the target device, includes: Receive a data access request sent by a terminal device, the data access request being used to request the reading of specified business data of the existing account; In response to the data access request, locate the specified business data in the local space; If the specified service data is found, the specified service data is returned to the terminal device; If the specified service data is not found, the index cluster is requested to return the routing information of the specified service data according to the stored target routing information, and the specified service data is obtained from the corresponding device based on the routing information returned by the index cluster, and the obtained specified service data is returned to the terminal device. After returning the specified service data to the terminal device, if the specified service data is being read for the first time, the unread amount described in the metadata is updated.

12. The method according to claim 11, characterized in that, The method further includes: Obtain the target routing information of the existing account from the target device or the index cluster; After obtaining the target routing information, the target routing information is stored in the local space.

13. The method according to claim 12, characterized in that, If the specified business data is not found, the method further includes: Based on the target routing information stored in the local space, determine the routing information of the specified service data, and request the corresponding device to return the specified service data based on the determined routing information; If an error code is received from the corresponding device, the step of requesting the index cluster to return the routing information of the specified business data is triggered.

14. The method according to claim 12, characterized in that, The target routing information in the local space is configured with a validity period; if the specified service data is not found, the method further includes: Calculate the duration for which the target routing information has been stored in the local space; Based on the stored duration and the valid duration, the target routing information in the local space is subjected to validity detection to obtain the detection result; If the detection result indicates that the target routing information in the local space has expired, then the step of requesting the index cluster to return the routing information of the specified business data is triggered.

15. The method according to claim 10, characterized in that, The metadata of the existing accounts includes a version number, and when the metadata changes, the corresponding version number monotonically increments; the method also includes: After receiving the metadata sent by the target device, search for the metadata of the existing account in the local space; If the corresponding metadata is found in the local space, the version number in the found metadata is determined as the first version number, and the version number in the received metadata is determined as the second version number. If the second version number is greater than the first version number, then the step of storing the received metadata in the local space is triggered.

16. A storage system expansion device, characterized in that, The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster is used to provide data read and write services for accounts. The cold cluster is used to store business data of accounts that are carried over from the hot cluster. The index cluster is used to store routing information of business data in the cold cluster. The device includes: The processing unit is configured to scan existing accounts from the local space of the target device when the target device in the hot cluster needs to be expanded, and to store the historical service data of each existing account in the local space; wherein, after the target device is expanded, at least one historical service data of the existing account is moved down to the cold cluster. The processing unit is further configured to determine metadata for describing the business data attributes of the existing account, and generate target routing information for the existing account. The target routing information is used to indicate the device where each historical business data of the existing account is located, and the device includes the target device. The output unit is used to migrate the metadata to the data migration device corresponding to the target device and store the target routing information in the index cluster; The processing unit is further configured to mark the existing account with a migration mark after migrating the metadata and storing the target routing information in the index cluster, so that the data migration-in device can provide data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster.

17. A storage system expansion device, characterized in that, The storage system includes a hot cluster, a cold cluster, and an index cluster. The hot cluster is used to provide data read and write services for accounts. The cold cluster is used to store business data of accounts that are carried over from the hot cluster. The index cluster is used to store routing information of business data in the cold cluster. The device includes: The receiving unit is used to receive metadata of existing accounts sent by the target device in the hot cluster. The metadata is used to describe the service data attributes of the existing accounts. The existing accounts are scanned from the local space of the target device when the target device needs to be expanded. Each historical service data of the existing account is kept stored in the local space when the target device needs to be expanded. After the target device is expanded, at least one historical service data of the existing account is moved to the cold cluster. The processing unit is used to store the received metadata in the local space, and after the target device marks the existing account with a migration mark, it provides data read and write services for the existing account on behalf of the target device based on the metadata and the target routing information in the index cluster. The target routing information is stored in the index cluster by the target device, and the target routing information is used to indicate the device where each historical business data of the existing account is located, including the target device.

18. A computer device, characterized in that, The method includes a processor and a memory, wherein the memory is used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1-9; or, when executed by the processor, the computer program implements the method as described in any one of claims 10-15.

19. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-9; or, when executed by a processor, the computer program implements the method as described in any one of claims 10-15.

20. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9; or, when executed by a processor, the computer program implements the method as described in any one of claims 10-15.

Images