Data storage method and system, device, storage medium, and program product

Abstract

A data storage method and system, a device, a storage medium, and a program product. The data storage method comprises: in response to a first data storage request sent by a client, a storage server separately caching target data to be stored in the first data storage request to a target memory and a first storage pool, wherein the target memory is a memory of the storage server, and a storage system where the storage server is located comprises the first storage pool; transferring the target data in the target memory to a second storage pool of the storage system; and deleting the target data in the target memory and the first storage pool, wherein the data read / write rate of the second storage pool is lower than that of the first storage pool. The data storage method reduces the frequency of read / write from / into the first storage pool and the second storage pool and the read / write throughput.

Description

Data storage methods, systems, devices, storage media and program products

[0001] This disclosure claims priority to Chinese Patent Application No. 202410927243.3, filed on July 10, 2024, entitled “Data Storage Method, System, Device, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, specifically to a data storage method, system, device, storage medium, and program product. Background Technology

[0003] In distributed storage systems, data is typically stored on disks using a multi-replica redundant storage method to meet reliability requirements. However, multi-replica redundant storage undoubtedly requires a significant amount of storage space. To save storage space, data is usually read from one disk and stored on another. To further save storage space, data is then read from that other disk, encoded, and saved again. This data storage method involves numerous disk read / write operations and high read / write throughput. Summary of the Invention

[0004] This disclosure addresses the issues of high read / write frequency and high read / write throughput in the aforementioned related technologies by providing a data storage method, system, device, storage medium, and program product.

[0005] The first aspect of this disclosure provides a data storage method applied to a storage service, the method comprising:

[0006] In response to the first data save request sent by the client, the target data to be saved in the first data save request is cached in the target memory and the first storage pool respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool;

[0007] The target data in the target memory is transferred to the second storage pool of the storage system; the data read / write rate of the second storage pool is lower than that of the first storage pool.

[0008] Delete the target data from the target memory and the first storage pool.

[0009] A second aspect of this disclosure provides a data storage system, which includes: a storage service client, a first storage pool, and a second storage pool; the data read / write rate of the second storage pool is lower than that of the first storage pool, and the storage service client is configured with a request processing service, a proxy service, and a storage service.

[0010] The request processing service is used to forward the first data save request sent by the client to the corresponding proxy service; the first data save request includes the target data to be saved.

[0011] The proxy service is used to cache the target data in the memory of the storage service and send a data caching request to the storage service; send a second data save request to the storage service based on the target data in memory; after the target data is transferred from memory to the second storage pool, delete the target data in memory and send a data deletion request to the storage service.

[0012] The storage service is used to cache target data in a first storage pool according to a data caching request; save target data in a second storage pool according to a second data saving request; and delete target data in the first storage pool according to a data deletion request.

[0013] A third aspect of this disclosure provides a data storage device for use in a storage service, the device comprising:

[0014] The caching module is used to respond to the first data save request sent by the client and cache the target data to be saved in the first data save request to the target memory and the first storage pool respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool;

[0015] The transfer module is used to transfer target data from the target memory to the second storage pool of the storage system; the data read / write rate of the second storage pool is lower than that of the first storage pool.

[0016] The deletion module is used to delete target data from the target memory and the first storage pool.

[0017] A fourth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method as described in the first aspect above.

[0018] A fifth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first aspect above.

[0019] A sixth aspect of this disclosure provides a computer program product including a computer program that is executed by a processor to implement the method described in the first aspect above.

[0020] Based on the data storage method provided in the first aspect above, this disclosure has at least the following beneficial effects or advantages:

[0021] In this embodiment, target data from the client is cached in a first storage pool within the storage system. Since the read / write rate of the second storage pool is lower than that of the first storage pool, the low latency of the first storage pool ensures fast and efficient data write-to-disk. Alternatively, the target data is cached in the storage service's memory and then transferred to the second storage pool. This process requires only one write operation to each storage pool, eliminating the need for multiple read / write operations. This significantly reduces the number of read / write operations to both storage pools. In other words, by using a dual-write method involving both the first and second storage pools, fast and efficient data write-to-disk is ensured while reducing the number of read / write operations and the overall throughput of the first and second storage pools.

[0022] The above description is only an overview of the technical solution of this disclosure. In order to better understand the technical means of this disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other objects, features and advantages of this disclosure more obvious and understandable, specific embodiments of this disclosure are given below. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0024] Figure 1 is a schematic diagram of the data storage process in related technologies;

[0025] Figure 2 is a schematic diagram of an application scenario of a data storage method provided in an embodiment of this disclosure;

[0026] Figure 3 is a first flowchart of a data storage method provided in an embodiment of this disclosure;

[0027] Figure 4 is a second flowchart of a data storage method provided in an embodiment of this disclosure;

[0028] Figure 5 is a schematic diagram of a data storage method provided in an embodiment of this disclosure;

[0029] Figure 6 is a third flowchart of a data storage method provided in an embodiment of this disclosure;

[0030] Figure 7 is a schematic diagram of the multiple services provided in the embodiments of this disclosure;

[0031] Figure 8 is a schematic diagram of a data storage device provided in an embodiment of this disclosure;

[0032] Figure 9 is a schematic diagram of a first composition of a data storage system provided in an embodiment of this disclosure;

[0033] Figure 10 is a schematic diagram of a second composition of a data storage system provided in an embodiment of this disclosure;

[0034] Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0035] Figure 12 is a schematic diagram of the structure of a storage medium provided in an embodiment of this disclosure. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination," etc.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0040] In large-scale data storage scenarios, storing data in multiple copies undoubtedly wastes a lot of storage space. For example, in a three-copy storage method, the number of data copies is 3, meaning three copies of the data are stored, and it is permissible for any two copies of the data to be corrupted or lost, but the required storage capacity is three times that of storing a single copy of the data. To save storage space, related technologies commonly use erasure coding (EC coding) for data storage. In EC coding, the data is split into k data blocks and m parity blocks, and the data redundancy ratio is (k+m) / k. Here, k is an integer greater than 1, and m is an integer greater than or equal to 1, such as k=8, m=3, etc. As shown in Figure 1-1, for a set of user data to be stored, the first storage method in related technologies is to first write three copies of the user data to a solid state disk (SSD). Because solid-state drives (SSDs) have small storage capacity and high cost, the three copies of data from the SSD will be migrated to a hard disk drive (HDD). To save storage space, a copy of the user data will be read from the HDD, EC encoded, and the encoded data (i.e., 8 data blocks and 3 parity blocks) will be saved to the HDD. In this data storage method, one data write operation and one data read operation are performed on the SSD; two data write operations and one data read operation are performed on the HDD. Both the SSD and HDD have a high number of read / write operations, increasing their throughput. To reduce the throughput of the SSD and HDD, as shown in Figure 1-2, a second storage method in related technologies is to directly EC encode a copy of the user data and save the encoded data (i.e., 8 data blocks and 3 parity blocks) to the HDD. While this method reduces the throughput of the SSD and HDD, it requires a large amount of user data and insensitivity to latency. If the user data volume is relatively small, it takes a long time to accumulate enough data before it can be written to disk. During this process, if the device loses power, data loss will occur. To ensure data integrity even in the event of a power outage, the only solution is to populate the disk with smaller amounts of user data to guarantee fast data writing. However, populating the disk with too much data wastes storage space. Therefore, this storage method has poor versatility and is not suitable for all data storage scenarios.

[0041] Based on this, this disclosure provides a data storage method. Figure 2 is a schematic diagram of an application scenario of the data storage method provided by this disclosure. As shown in Figure 2, the scenario includes a storage service client, a client, a first storage pool, and a second storage pool. The storage service client can be one or more services, which can be set in various storage nodes of the storage system. These storage nodes can be terminal devices such as desktop computers, laptops, home appliances, and vehicle terminals. The storage service client can also be one or more servers, which can be physical servers or cloud servers, etc.; these servers can be storage nodes of the storage system. Among the storage nodes included in the storage system where the storage service client is located, some storage nodes can have a first storage area, some storage nodes can have a second storage area, and some storage nodes can have both a first storage area and a second storage area simultaneously. Each first storage area constitutes a first storage pool, and each second storage area constitutes a second storage pool. The storage system can be a distributed storage system, such as an object storage system or a file storage system. The client can be set in a user device. The client can be an application (APP), a mini-program embedded in an application, or a web application, etc. User devices can be mobile phones, tablets, desktop computers, laptops, wearable devices, etc. Figure 2 illustrates the example with a physical server as the storage service provider (illustrated using a single physical server) and a desktop computer as the user device. It should be understood that the storage service provider and user device are not limited to these examples.

[0042] Specifically, any storage service client can communicate with the client and perform storage processing on the target data to be saved in the first data save request sent by the client. This storage service client can maintain data saving strategies, data deletion strategies, data processing strategies, data query strategies, etc., and can also access the first and second storage pools of its storage system. The data saving strategy instructs the caching of the target data to be saved in the first storage pool and the storage service client's memory, respectively, the transfer of the target data in memory to the second storage pool, and the saving of the target data's query data. The data deletion strategy instructs the deletion of the target data in memory and the cached target data in the first storage pool after the target data is transferred to the second storage pool. The data processing strategy instructs the encoding algorithm and encoding parameters used to encode the target data during the storage process. The data query strategy instructs how to query the target data based on the first query data after caching it, and how to query the target data based on the second query data after transferring it. Therefore, during the storage of target data, the target data from the client is cached in the first storage pool of the storage system. Since the data read / write rate of the second storage pool is lower than that of the first storage pool, the low latency of the first storage pool ensures that the target data is quickly and effectively written to disk. Alternatively, the target data is cached in the storage service's memory and then transferred to the second storage pool based on the data in memory. This process requires only one write operation to each of the first and second storage pools, eliminating the need for multiple read / write operations. This significantly reduces the number of read / write operations to both pools. In other words, by using a dual-write approach involving the first storage pool and memory, the system ensures fast and effective data writing to disk while reducing the number of read / write operations and the overall read / write throughput of both pools.

[0043] It should be understood that Figure 2 is merely a schematic representation of an application scenario of the data storage method involved in this disclosure, and does not constitute a limitation on the technical solution of this disclosure. In other embodiments, the application scenario of the data storage method involved in this disclosure may include more or fewer components.

[0044] The technical solutions of this disclosure and how they solve the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0045] Figure 3 is a flowchart of a data storage method provided in an embodiment of this disclosure. The method shown in Figure 3 can be executed by the storage service client in Figure 2. As shown in Figure 3, the method includes steps 101 to 103.

[0046] Step 101: In response to the first data save request sent by the client, cache the target data to be saved in the first data save request into the target memory and the first storage pool respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool.

[0047] Step 102: Transfer the target data in the target memory to the second storage pool of the storage system. The data read / write rate of the second storage pool is lower than that of the first storage pool.

[0048] Step 103: Delete the target data in the target memory and the first storage pool.

[0049] In some implementations, the storage capacity of the first storage pool is smaller than that of the second storage pool, and the cost of the first storage pool is higher than that of the second storage pool. The first storage pool may be, for example, a solid-state drive (SSD) pool, and the second storage pool may be, for example, a hard disk drive (HDD) pool; correspondingly, the aforementioned first storage area may be a solid-state drive, and the second storage area may be a hard disk drive. This disclosure does not specifically limit the specific types of the first and second storage pools; they can be set as needed in practical applications.

[0050] Specifically, when a user needs to save target data, they can submit the target data to be saved through a client on their user device. The client responds to the user's submission by sending a first data save request to the storage service provider based on the obtained target data. Considering the high throughput, low latency, and data retention characteristics of solid-state drives (SSDs), to ensure the target data is quickly and effectively written to disk, the storage service provider, in response to the first data save request, caches the target data in the first storage pool of the storage system. Considering the small capacity and high cost of SSDs, to save storage space, reduce costs, and decrease SSD throughput, the storage service provider also caches the target data in its own memory and performs a transfer process based on the target data in memory to transfer the target data to the second storage pool of the storage system. Since retaining the target data in memory and the first storage pool is less meaningful after the data is transferred to the second storage pool, the storage service provider deletes the target data cached in memory and the first storage pool to promptly release storage space.

[0051] In some implementations, the storage system where the aforementioned storage service client is located can provide object storage service (OSS), that is, the target data can also be called the target object, which can be a file, image, audio or video, etc.

[0052] In some embodiments, caching the target data in memory and caching the target data in the first storage pool can be performed simultaneously; alternatively, the operation of caching the target data in memory can be performed first, followed by the operation of caching the target data in the first storage pool; or the operation of caching the target data in the first storage pool can be performed first, followed by the operation of caching the target data in memory. This disclosure does not impose specific limitations on these methods.

[0053] In the data storage method provided in this embodiment, target data from the client is cached in a first storage pool included in the storage system. Since the data read / write rate of the second storage pool is lower than that of the first storage pool, the low latency of the first storage pool ensures that the target data is quickly and effectively written to disk. Alternatively, the target data is cached in the storage service client's memory and then transferred to the second storage pool based on the target data in memory. This process only requires one data write operation to each of the first and second storage pools, eliminating the need for multiple data read / write operations. This significantly reduces the number of read / write operations to the first and second storage pools. In other words, by using a dual-write method involving the first storage pool and memory, the number of read / write operations and the read / write throughput of the first and second storage pools are reduced while ensuring fast and effective data writing to disk.

[0054] To conserve storage space in the first storage pool, in some implementations, the target data is encoded and then cached in the first storage pool. Specifically, as shown in Figure 4, step 101 may include steps 1011 to 1013:

[0055] Step 1011: In response to the first data save request sent by the client, cache the target data to be saved in the first data save request to the target memory; wherein, the target memory is the memory of the storage service client;

[0056] To facilitate quick retrieval of target data from target memory later, in some implementations, the target data and its data identifier can be associated and cached in target memory. Alternatively, the target data can be cached in target memory, and the association between the storage location of the target data in target memory and its data identifier can be recorded. The data identifier of the target data can be obtained by the storage service provider from the first data save request, or it can be assigned to the target data by the storage service provider.

[0057] Step 1012: Perform a first encoding process on the target data to obtain first encoded data; the first encoded data includes a first number of first encoded blocks;

[0058] In some implementations, the target data can be processed using the EC encoding algorithm and first encoding parameters to obtain first encoded data. The first encoding parameters include k1 and m1, where k1 is an integer greater than 1 and m1 is an integer greater than or equal to 1. Correspondingly, the first quantity = k1 + m1, meaning the first encoded block includes k1 first data blocks and m1 first parity blocks. Specifically, the target data can be split according to the first encoding parameters to obtain k1 first data blocks; and m1 first parity blocks can be generated based on the first encoding parameters and the k1 first data blocks to obtain the first encoded data. For a detailed explanation of the EC encoding process, please refer to relevant technologies; it will not be elaborated upon in this disclosure.

[0059] Step 1013: Cache the first encoded data to the first storage pool; the storage system where the storage service client is located includes the first storage pool.

[0060] By performing a first encoding operation on the target data and caching the resulting first encoded data to the first storage pool, the storage space required to cache the target data can be reduced, thus saving storage space in the first storage pool.

[0061] Furthermore, to reduce the storage space required for the second storage pool to store the target data, in some embodiments, the target data can be encoded and then saved to the second storage pool. Specifically, as shown in Figure 4, step 102 may include steps 1021 to 1023:

[0062] Step 1021: Read target data from target memory;

[0063] Specifically, when the target data is associated with the data identifier of the target data and cached in the target memory in step 1011, step 1021 may include: querying the associated target data stored in the target memory according to the data identifier of the target data, and reading the queried target data.

[0064] When the target data is cached in the target memory in step 1011 and the association between the storage location of the target data in the target memory and the data identifier of the target data is recorded, step 1021 may include: querying the associated third storage location in the recorded association based on the data identifier of the target data, and reading the target data at the third storage location in the target memory.

[0065] Step 1022: Perform a second encoding process on the target data to obtain second encoded data; the second encoded data includes a second number of second encoded blocks;

[0066] In some implementations, the target data can be subjected to a second encoding process based on the EC encoding algorithm and a second encoding parameter to obtain second encoded data. The second encoding parameter includes k2 and m2, where k2 is an integer greater than 1 and m2 is an integer greater than or equal to 1. Correspondingly, the second quantity = k2 + m2, meaning the first encoded block includes k2 second data blocks and m2 second check blocks. The process of the second encoding process is the same as the process of the first encoding process described above, as can be found in the relevant description above. The second encoding parameter may or may not be the same as the aforementioned first encoding parameter.

[0067] Step 1023: Save the second encoded data to the second storage pool of the storage system.

[0068] By performing a second encoding operation on the target data and caching the resulting second encoded data in the second storage pool, the storage space required to store the target data can be reduced, thus saving storage space in the second storage pool.

[0069] It should be noted that Figure 4 is for illustrative purposes only and not for limitation. The first and second encoding processes described above can both be performed, or only one can be performed. That is, in some embodiments, only the first encoding process can be performed, and the original target data can be saved to the second storage pool. Alternatively, only the second encoding process can be performed, and the original target data can be cached in the first storage pool. This can be configured as needed in practical applications.

[0070] Taking the execution of both the first and second encoding processes described above, with the first and second encoding parameters differing, as an example, the advantages of the data storage method disclosed herein compared to the first and second storage methods in the aforementioned related technologies are further illustrated. For instance, the first encoding parameters include k1 = 4 and m1 = 2, and the second encoding parameters include k2 = 8 and m2 = 3. Both storage methods in the related technologies use the second encoding parameters. Therefore:

[0071] In the first storage method of the related technology, for solid-state drives (SSDs), three copies of the target data are stored, one data write operation is performed, and one data read operation is performed. For hard disk drives (HDDs), since one copy of the target data is EC encoded using the second encoding parameter, the data redundancy ratio is (8+3) / 8 = 1.375, meaning the resulting encoded data is equivalent to 1.5 copies of the target data. In other words, the HDD stores 3 copies of the target data + 1.375 copies of the target data = 4.375 copies of the target data, and one data read operation and two data write operations are performed.

[0072] In the second storage method of the related technology, since the target data is EC encoded by the second encoding parameter and then saved to the mechanical hard disk, the data redundancy ratio is (8+3) / 8 = 1.375. Therefore, 1.375 copies of the target data are stored in the mechanical hard disk, and a data write operation is performed once.

[0073] In the data storage method provided in this disclosure, as shown in Figure 5, for the first storage pool, since EC encoding is performed on one copy of target data using the first encoding parameter, the data redundancy ratio is (4+2) / 4 = 1.5, that is, the obtained first encoded data is equivalent to 1.5 copies of target data. Therefore, the first storage pool stores 1.5 copies of target data and performs one data write operation. For the second storage pool, 1.375 copies of target data are stored and one data write operation is performed.

[0074] As can be seen, compared to the first storage method in related technologies, the data storage method provided in this disclosure significantly reduces the amount of data stored in both the first and second storage pools, and also reduces the number of read / write operations. This greatly saves storage space and reduces the number of read / write operations and write throughput in both the first and second storage pools. Compared to the second storage method in related technologies, although the amount of data stored in the second storage pool is the same and the number of read / write operations is the same, the second storage method has requirements regarding data size and latency, thus limiting its application scenarios. The data storage method provided in this disclosure, however, uses the first storage pool to cache target data. Therefore, based on the low latency of the first storage pool, it can quickly write data to disk regardless of its size, meaning there are no requirements regarding data size, and it is applicable even with high latency requirements, making it more universally applicable.

[0075] Furthermore, it is understandable that in distributed storage scenarios, the storage service client and the first and second storage pools may not be on the same device. Therefore, data storage may require data transfer across devices and data centers via network. Since the data storage method provided in this disclosure significantly reduces the amount of data stored in both the first and second storage pools, the amount of data transferred over the network is also significantly reduced, thereby reducing the network bandwidth required for cross-device and cross-data center transmissions and lowering network transmission latency.

[0076] In practical applications, distributed storage typically employs a partitioned storage method, with a partition server providing services such as user data read / write and background data recycling. There is a one-to-many correspondence between the partition server and the partitions, meaning one partition server provides data read / write services to multiple partitions. This approach places significant pressure on the partition server, easily leading to slow access speeds, service unavailability, and even data loss or corruption. This issue can also be termed a hotspot problem. In related technologies, one approach to address this issue is to distribute user data evenly across all partitions. When deleting user data requires garbage collection across multiple partitions to truly free up space, which is time-consuming and inefficient. Another approach is to split based on pressure, such as splitting a partition into multiple partitions. However, splitting requires careful consideration of pressure and data volume, and the splitting time can be at least minutes, or even hours, resulting in significant time consumption. Therefore, this disclosure proposes the role of a proxy service to distribute the pressure on the partition server. Specifically, in some implementations, the storage service client may include a proxy server and a partitioning service.

[0077] Accordingly, in step 101 above, caching the target data to be saved in the first data save request to the target memory and the first storage pool respectively may include: caching the target data to the target memory and the first storage pool through a proxy service.

[0078] The aforementioned step 102, which involves transferring the target data in the target memory to the second storage pool of the storage system, may include: transferring the target data in the target memory to the second storage pool through a proxy service.

[0079] Considering that in practical applications, users may have data query needs both after the target data is cached in the first storage pool and after it is transferred to the second storage pool, the method may further include: obtaining first query data and second query data through a proxy service, and sending the first query data and second query data to a partitioning service. The first query data is used to query the target data stored in the first storage pool, and the second query data is used to query the target data stored in the second storage pool. Furthermore, the partitioning service provides the client with a target data query service based on either the first query data or the second query data.

[0080] In other words, the first data save request from the client is handled by the proxy service, which provides the data write service externally. The partition service, however, only provides the client with the target data query service based on the first or second query data sent by the proxy service; that is, the partition service provides the corresponding partition's data read service externally. Therefore, the proxy service distributes the service load of the partition service, ensuring its effective operation and fast access.

[0081] In some implementations, as shown in Figure 6, the storage service client may also include a Chunk Server, which can be used to manage the organization and storage of standalone data and provide data read, write, and delete services to the outside world.

[0082] Accordingly, the aforementioned caching of target data to the first storage pool via the proxy service, and the transfer of target data in target memory to the second storage pool via the proxy service, may include:

[0083] Send data caching requests to the storage service through the proxy service, and send a second data saving request to the storage service through the proxy service;

[0084] Based on the data caching request, the target data is cached in the first storage pool using the storage service; based on the second data saving request, the target data is saved in the second storage pool.

[0085] More specifically, as shown in Figure 6, the target data can be processed by a proxy service to perform a first encoding process, resulting in first encoded data. The proxy service then sends a cache request to the storage service based on the first encoded data. The storage service caches the first encoded data in a first storage pool according to the cache request. Additionally, the target data can be read from the target memory by the proxy service and processed by a second encoding process to obtain second encoded data. The proxy service then sends a second data save request to the storage service based on the second encoded data. The storage service saves the second encoded data to a second storage pool according to the second data save request.

[0086] Since the first encoded data is cached in the first storage pool through the storage service, and the second encoded data is saved in the second storage pool through the storage service, the storage service knows the storage location of the first encoded data in the first storage pool and the storage location of the second encoded data in the second storage pool. To facilitate subsequent data query processing, in some embodiments, as shown in Figure 6, after caching the first encoded data in the first storage pool through the storage service, the storage service also generates first query data based on the storage location of the first encoded data in the first storage pool and sends the first query data to the proxy service. Similarly, after saving the second encoded data in the second storage pool through the storage service, the storage service also sends the storage location information of the second encoded data in the second storage pool to the proxy service; the proxy service generates second query data based on the first query data and the storage location information, sends the second query data to the storage service, the storage service saves the second query data, and sends a fourth success message to the proxy service. In some embodiments, the storage service can save the second query data in the second storage pool. The proxy service can send the second query data and the data identifier of the target data to the storage service; correspondingly, the storage service associates the second query data and the data identifier of the target data and saves them in the second storage pool.

[0087] The first query data may include the mapping relationship between the block identifier of each first data block and its storage location in the first storage pool, and the mapping relationship between the block identifier of each first verification block and its storage location in the first storage pool. The storage location information may include the mapping relationship between the block identifier of each second data block and its storage location in the second storage pool, and the mapping relationship between the block identifier of each second verification block and its storage location in the second storage pool. The second query data may include the mapping relationship between the block identifier of each first data block and its storage location in the second storage pool, and the mapping relationship between the block identifier of each second verification block and its storage location in the second storage pool.

[0088] In some implementations, the first coded data and the second coded data are stored in corresponding files, and these files are saved to the corresponding first and second storage pools. Accordingly, the storage location information of the first query data and the second coded data in the second storage pool can be in the form of "block identifier: file information, starting position, data length". Here, file information can be a file name, file identifier, etc., the starting position can also be called an address offset, and the data length is the data length of the data block or check block corresponding to the block identifier. In some implementations, the first query data can be a series of records. To improve the management efficiency of the second query data and save the input / output resources required for transmitting the second query data, the second query data can be in the form of a file containing a mapping relationship generated based on the first query data and the storage location information. Each mapping relationship can correspond to multiple records in the first query data.

[0089] As an example, the first data block includes first data block 1 and first data block 2, where first data block 1 corresponds to bytes 0 to 50 in the target data, and first data block 2 corresponds to bytes 51 to 100 in the target data; the block identifier of first data block 1 is 01, and the block identifier of first data block 2 is 02. Bytes 0 to 100 of the target data are located at storage location 1 at the beginning of the second storage pool. The second query data can then include the mapping relationship between the first and second data blocks and storage location 1, for example, denoted as "01, 02: file information, storage location 1, 100 megabytes". That is, this mapping relationship corresponds to two records in the first query data. It can be seen that compared to the first query data, the number of mapping relationships contained in the second query data is significantly reduced, which not only improves the maintenance efficiency of the second query data but also reduces the input / output resources required to transmit the second query data.

[0090] It should be noted that the above examples are for illustration only and not for limitation. The specific forms of the first and second query data can be set as needed in actual applications, and this disclosure does not impose specific limitations on them.

[0091] Corresponding to the above-mentioned storage service generating the first query data and the proxy service generating the second query data, the aforementioned acquisition of the first query data and the second query data through the proxy service may include: receiving the first query data sent by the storage service through the proxy service, and generating the second query data through the proxy service.

[0092] In some implementations, as shown in Figure 6, the storage service client may further include a request processing service. As shown in Figure 7, this request processing service may correspond to at least one proxy service and communicate with the client. That is, the request processing service can receive a first data save request sent by the client and send the first data save request to the corresponding proxy service.

[0093] Correspondingly, as shown in Figure 6, sending the first query data to the partition service can include: sending the first query data to the request processing service through a proxy service; and sending the first query data to the partition service through the request processing service. Sending the second query data to the partition service can include: sending the second query data to the partition service through a storage service.

[0094] As shown in Figure 7, by setting up a proxy service, the request processing service can send the first data save requests from each client to the corresponding proxy service, which then performs the storage processing of the target data. The request processing service only needs to send the first query data corresponding to the target data to the partition service, without having to send each first data save request directly to the partition service. Therefore, the proxy service distributes the service load of the partition service, ensuring its effective operation and fast access.

[0095] To facilitate the partitioning service's management of the first and second query data for the target data, in some implementations, a proxy service can send the data identifiers of the first query data and the target data to the request processing service, and the request processing service can send the data identifiers of the first query data and the target data to the partitioning service. Accordingly, the partitioning service associates and stores the data identifiers with the first query data.

[0096] Furthermore, to inform the request processing service of the saving result of the first query data, and to inform the user of the data saving result, as shown in Figure 6, in some embodiments, the partition service saves the received first query data and sends a third data saving request to the storage service. The storage service, based on the third data saving request, saves the first query data and sends a first success message indicating successful saving to the partition service. After receiving the first success message, the partition service can also send a second success message to the request processing service. After receiving the second success message, the request processing service can also send a third success message to the client. In some embodiments, the storage service can save the first query data to a first storage pool. The request processing service can send a third success message indicating successful data saving to the client based on the data identifier of the target data and the block identifiers of each first data block included in the first query data, so that the client can send a first data query request based on the block identifiers.

[0097] In practical applications, the client may send the first data query request before or after the target data has been transferred. To ensure effective querying of the target data, the aforementioned method of sending the second query data to the partition service via a proxy service can include sending the second query data to the storage service via the proxy service, sending a retrieval request to the storage service via the partition service, and receiving the second query data sent by the storage service. Correspondingly, upon receiving the second query data, the partition service deletes the first query data, as shown in Figure 6. Specifically, the partition service sends a retrieval request to the storage service based on the data identifier of the target data; the storage service queries the associated and stored second query data based on the data identifier in the retrieval request, and sends the second query data and the data identifier to the partition service; the partition service queries the associated and stored first query data based on the received data identifier and deletes the retrieved first query data.

[0098] It is evident that the partition service contains the first query data or has obtained the second query data from the storage service. The presence of the first query data indicates that the target data transfer is incomplete, requiring data query processing based on the first storage pool. The acquisition of the second query data indicates that the target data transfer is complete, requiring data query processing based on the second storage pool. Therefore, the aforementioned provision of target data query services to clients based on the first or second query data can include:

[0099] If the transfer of target data to the second storage pool is not completed, and a first data query request is received from the client, then according to the target block identifier in the first data query request, the corresponding first storage location is queried in the first query data, and the target data block corresponding to the target block identifier is queried from the first storage location in the first storage pool.

[0100] If the first data query request is received after the target data has been transferred to the second storage pool, the corresponding second storage location is queried in the second query data according to the target block identifier, and the target data block corresponding to the target block identifier is queried from the second storage location in the second storage pool.

[0101] Assuming the target data transfer is complete, specifically as shown in Figure 6, the request processing service receives a first data query request from the client and sends an address query request to the partitioning service based on the target block identifier in the first data query request. The partitioning service then queries the corresponding second storage location in the second query data based on the target block identifier in the address query request and sends the second storage location to the request processing service. The request processing service then sends a second data query request to the storage service based on the second storage location. The query service then queries the target data block corresponding to the target block identifier at the second storage location in the second storage pool based on the second storage address in the second data query request and sends the target data block to the request processing service. Finally, the request processing service sends the target data block to the client.

[0102] It should be noted that Figure 6 is for illustration only and not for limitation. Step 7 may be the storage service sending a sixth success message to the proxy service, which carries the first query data; step 8 may be the proxy service sending a seventh success message to the request processing service, which carries the first query data. The execution order of some operations can be interchanged or executed simultaneously (e.g., execute steps 4 and 5 first, then execute step 3; or execute steps 3 and 4 simultaneously; steps 14 to 24 may be completed by background tasks, and steps 14 to 24 can be executed regardless of the state of steps 8 to 12, that is, after step 7, the proxy service can execute step 14 after receiving the first query data, etc.), and may also include more operations (e.g., data query operations based on the first storage pool, that is, before step 20, the request processing service receives the third data query request sent by the client, and sends a location to the partition service according to the target block identifier in the third data query request). Address query request. Based on the target block identifier in the address query request, the partition service queries the corresponding first storage location in the first query data and sends the first storage location to the request processing service. Based on the first storage location, the request processing service sends a fourth data query request to the storage service. Based on the first storage address in the fourth data query request, the query service queries the target data block corresponding to the target block identifier at the first storage location in the first storage pool and sends the target data block to the request processing service; the request processing service then sends the target data block to the client. (That is, after the storage service saves the first query data in step 11, it can provide data query services, or even fewer operations.)

[0103] Therefore, by performing different operations through different services and matching them with each other, not only can the target data be stored and retrieved quickly and effectively, but the access pressure can also be effectively distributed to ensure the effective provision of data read and write services to the outside world.

[0104] As mentioned above, the data deletion service is provided through the storage service. Correspondingly, as shown in Figure 6, deleting the target data in the first storage pool can include:

[0105] The proxy service sends a data deletion request to the storage service based on the data identifier of the target data; the storage service deletes the target data corresponding to the data identifier in the first storage pool based on the data deletion request, and sends a fifth success message indicating successful deletion to the proxy service.

[0106] Specifically, the aforementioned cache request may include a data identifier for the target data. After the storage service generates the first query data, it can associate and save the data identifier with the first query data. Correspondingly, upon receiving a data deletion request, it can query the associated and saved first query data based on the data identifier in the data deletion request, read the storage address from the first query data, and delete the cached target data at that storage address in the first storage pool.

[0107] Therefore, after the target data is transferred, timely deletion of the target data in the first storage pool can promptly release the storage space of the first storage pool.

[0108] It should be noted that the deletion process for target data in the first storage pool is not limited to the process described above. A garbage collection mechanism can also be pre-configured, and the target data in the first storage pool can be deleted when the deletion conditions are met based on the garbage collection mechanism. Furthermore, the operation of deleting target data in the first storage pool is not limited to being performed by the storage service; it can also be performed by the proxy service. In the event of an anomaly in the proxy service, the target data in the first storage pool can also be deleted by the partition service.

[0109] In some implementations, the aforementioned partitioning service also has the capability to dispatch proxy services. Accordingly, before receiving the first data saving request sent by the client through the request processing service, the method may further include:

[0110] The request processing service sends a proxy service allocation request to the partition service; the partition service then allocates the corresponding proxy service to the request processing service based on the proxy service allocation request.

[0111] The process of allocating corresponding proxy services to the request processing service based on the proxy service allocation request can include: determining the target number of proxy services to be allocated based on the proxy service allocation request, and allocating the target number of proxy services to the request processing service. Specifically, it involves determining whether the proxy service allocation request contains a target number; if so, obtaining the target number from the proxy service allocation request; if not, setting the default number as the target number; and allocating the target number of proxy services to the request processing service.

[0112] It should be noted that a proxy service can be repeatedly assigned to multiple request processing services; that is, one request processing service corresponds to at least one proxy service, and one proxy service corresponds to at least one request processing service. Accordingly, a request processing service can send a first data saving request along with its second service identifier to a proxy service, so that the proxy service will send the obtained first query data to the request processing service corresponding to the second service identifier.

[0113] In some implementations, the encoded target data is stored in a file, and the file is stored in both a first storage pool and a second storage pool. To ensure effective data storage, the partitioning service also maintains file information for each file corresponding to its target partition, and after allocating a proxy service to the request processing service, establishes a correspondence between the proxy service and the file information, and sends the file information to the corresponding proxy service. Accordingly, the aforementioned cache request and second data storage request also include this file information. The storage service saves the first encoded data to the file corresponding to the file information in the cache request and saves that file to the first storage pool. Similarly, the storage service saves the second encoded data to the file corresponding to the file information in the second data storage request and saves that file to the second storage pool.

[0114] Therefore, by maintaining the correspondence between each proxy service and the file, the partition service ensures that the target data is accurately stored in the target partition corresponding to the partition service.

[0115] Considering that the read / write load of data in each partition is usually variable in practical applications, in order to ensure that the target partition can effectively provide data read / write services, in some implementations, the method may further include:

[0116] Based on the first query data, the partition service determines the first access status information of the target partition corresponding to the partition service; wherein, the first access status information is used to characterize the size of the read and write load of the target partition; if the third number of proxy services in the target partition does not match the first access status information, then the proxy services in the target partition are added or reduced.

[0117] In some implementations, the first query data may further include the block size of each coded block. Correspondingly, a fifth quantity of each received first query data can be counted through a partition service, and the first total data volume of each target data corresponding to the first query data can be counted based on the block size included in the first query data; first access status information is determined based on the fifth quantity and the first total data volume. The first access status information may include load level information determined based on the fifth quantity and the first total data volume. Furthermore, based on the load level information in the first access status, the corresponding first target quantity range is obtained from the first correspondence between the load level information and the quantity range of proxy services. A third quantity of proxy services already allocated in the target partition is counted, and the relationship between the third quantity and the first target quantity range is determined. If the third quantity is less than the lower limit of the first target quantity range, it is determined that the third quantity of proxy services in the target partition does not match the first access status information, and a proxy service is created to allocate a new proxy service to the request processing service when a proxy service allocation request is received from the request processing service. If the third quantity is within the first target quantity range, it is determined that the third quantity of proxy services in the target partition matches the first access status information. If the third quantity is greater than the upper limit of the first target quantity range, it is determined that the third quantity of proxy services in the target partition does not match the first access status information, and the proxy service to be removed is determined and removed.

[0118] In some implementations, when the request processing service sends the first query data sent by the proxy service to the partition service, it can send the first query data along with the first service identifier of the proxy service to the partition service. The partition service can save the correspondence between the reception time of the first query data and the first service identifier. Accordingly, determining the proxy service to be removed may include: for each proxy service, querying the last saved reception time according to the first service identifier of the proxy service; if it is determined that the duration of the last reception time from the current time is greater than a duration threshold, then the corresponding proxy service is determined to be a proxy service to be removed.

[0119] When the third number of proxy services in the target partition does not match the first access status information, it indicates that the data read / write load of the target partition is high, and the existing proxy services are insufficient to handle this high load. Alternatively, it indicates that the data read / write load in the target partition is low, and the existing proxy services are all handling small read / write loads, or some existing proxy services are idle, resulting in resource waste. Therefore, when it is determined that the third number of proxy services in the target partition does not match the first access status information, adding or reducing the number of proxy services in the target partition can avoid excessive access pressure on proxy services and prevent idle proxy services, thus ensuring the effective operation of the storage service.

[0120] It should be noted that the method for determining the first access status information and the specific process for determining whether the third number of proxy services in the target partition matches the first access status information are not limited to the aforementioned methods, and can be set as needed in actual applications.

[0121] Furthermore, as mentioned above, the request processing service can correspond to at least one proxy service. When the request processing service corresponds to multiple proxy services, in order to improve the efficiency of saving the target data, in some implementations, the aforementioned sending of the first data saving request to the corresponding proxy service through the request processing service may include:

[0122] The request processing service determines the target proxy service with the smallest latency based on the latency information of each proxy service within a preset time period. The latency information represents the time between the request processing service sending the data to be stored to the proxy service and receiving the first query data corresponding to the data to be stored returned by the proxy service.

[0123] The request processing service sends the first data saving request to the target proxy service.

[0124] Since latency information can characterize the request processing performance of a proxy service, sending the first data save request to the target proxy service with the lowest latency ensures that the first data save request is processed by the proxy service with the best current performance, which helps improve data storage efficiency.

[0125] Furthermore, in order to avoid resource waste while ensuring that data can be stored effectively, in some implementations, the method may also include:

[0126] The request processing service saves the request based on the first data and determines the second access status information of the request processing service; the second access status information is used to characterize the size of the send and receive load of the request processing service.

[0127] If the fourth number of proxy services corresponding to the request processing service does not match the second access status information, a proxy service allocation request is sent to the partition service.

[0128] By partitioning the service and allocating requests to proxy services, the corresponding proxy services for request processing services can be added or reduced.

[0129] In some implementations, the request processing service can count the sixth number of received first data saving requests and, based on each first data saving request, count the second total amount of corresponding target data; and, based on the sixth number and the second total amount of data, determine second access status information. The second access status information may include load level information determined based on the sixth number and the second total amount of data. Furthermore, based on the load level information in the second access status, the corresponding second target quantity range is obtained from the second correspondence between the load level information and the quantity range of proxy services, determining the fourth number of proxy services currently corresponding to the request processing service, and determining the relationship between the fourth number and the second target quantity range. If the fourth number is less than the lower limit of the second target quantity range, it is determined that the fourth number of proxy services corresponding to the request processing service does not match the second access status information, and a proxy service allocation request is sent to the partitioning service; the partitioning service allocates a new proxy service to the request processing service based on the proxy service allocation request. If the fourth number is within the second target quantity range, it is determined that the fourth number of proxy services corresponding to the request processing service matches the second access status information. If the fourth quantity exceeds the upper limit of the target quantity range, it is determined that the fourth quantity of the proxy service corresponding to the request processing service does not match the second access status information, and a proxy service removal request is sent to the partition service. The partition service then identifies the proxy service to be removed from the proxy services corresponding to the request processing service and removes it. In other words, the aforementioned proxy service allocation request can be either a proxy service allocation request or a proxy service removal request. The process by which the partition service determines the proxy service to be removed can be found in the previous description, and will not be repeated here.

[0130] Therefore, by determining whether the fourth number of the corresponding proxy services matches the second access status information through the request processing service, and sending a proxy service allocation request to the partition service when they do not match, the request processing service can effectively process each first data saving request received by the request processing service, thereby avoiding idle proxy services and thus avoiding resource waste.

[0131] Considering that in practical applications, various factors such as network issues may cause abnormal access to the proxy service, in order to ensure that the first data saving request can be processed smoothly, in some implementations, the method may further include:

[0132] If the request processing service determines that any of the corresponding proxy services is accessing abnormally, an exception message is sent to the partition service.

[0133] If the partition service determines, based on the exception message, that the proxy service with access issues meets the removal criteria, then the proxy service with access issues will be removed.

[0134] Specifically, if sending a first data save request to any of its corresponding proxy services through the request processing service fails, or if the first query data sent by the proxy service is not received within a first preset time period after sending the first data save request, it is determined that the proxy service access is abnormal, and an abnormality prompt message is sent to the partition service according to the first service identifier of the proxy service. The partition service obtains the first service identifier from the abnormality prompt message and counts the seventh number of abnormality prompt messages containing the first service identifier received within a second preset time period; if it is determined that the seventh number is greater than the number threshold, the proxy service corresponding to the first service identifier is removed.

[0135] Understandably, when the number of seventh requests exceeds the threshold, it indicates that most of the request processing services corresponding to the proxy service have reported that the proxy service is abnormal. At this time, removing the proxy service not only ensures the rationality of the removal, but also avoids the risk of data saving failure caused by the request processing service sending the first data saving request to the proxy service.

[0136] Corresponding to the embodiments of the aforementioned data storage methods, this disclosure also provides embodiments of a data storage device. FIG8 is a schematic structural diagram of a data storage device according to an exemplary embodiment, which can be configured in the storage service client shown in FIG1. ​​The device is used to execute the data storage method provided in any of the above embodiments. As shown in FIG8, the data storage device includes:

[0137] The caching module 201 is used to respond to the first data save request sent by the client and cache the target data to be saved in the first data save request to the target memory and the first storage pool respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool.

[0138] The transfer module 202 is used to transfer the target data in the target memory to the second storage pool of the storage system. The data read and write rate of the second storage pool is lower than that of the first storage pool.

[0139] The deletion module 203 is used to delete the target data in the target memory and the first storage pool.

[0140] In some implementations, the cache module 201 is specifically used for:

[0141] The target data is subjected to a first encoding process to obtain first encoded data; the first encoded data includes a first number of first encoded blocks.

[0142] The first encoded data is cached in the first storage pool.

[0143] In some implementations, the transfer module 202 is specifically used for:

[0144] Read target data from target memory;

[0145] The target data is subjected to a second encoding process to obtain second encoded data; the second encoded data includes a second number of second encoded blocks.

[0146] The second encoded data is saved to the second storage pool of the storage system.

[0147] In some implementations, the storage service client includes a proxy service and a partitioning service, and the device also includes an acquisition module and a query module;

[0148] Correspondingly, the caching module 201 is specifically used to cache target data to target memory and the first storage pool through the proxy service, and to transfer target data in target memory to the second storage pool through the proxy service.

[0149] The acquisition module is used to acquire first query data and second query data through the proxy service, and send the first query data and second query data to the partition service; the first query data is used to query the target data stored in the first storage pool, and the second query data is used to query the target data stored in the second storage pool.

[0150] The query module is used to provide the client with a query service for target data based on the first query data or the second query data through the partition service.

[0151] In some implementations, the first query data includes a mapping relationship between the block identifier of a first data block of the target data and the storage location of the first data block in a first storage pool; the second query data includes a mapping relationship between the block identifier and the storage location of the first data block in a second storage pool; correspondingly, the query module is specifically used for:

[0152] If the transfer of target data to the second storage pool is not completed, and a first data query request is received from the client, the first storage location corresponding to the target block identifier in the first data query request is queried in the first query data, and the target data block corresponding to the target block identifier is queried from the first storage location in the first storage pool.

[0153] If the first data query request is received after the target data has been transferred to the second storage pool, the corresponding second storage location is queried in the second query data according to the target block identifier, and the target data block corresponding to the target block identifier is queried from the second storage location in the second storage pool.

[0154] In some embodiments, the apparatus further includes a first determining module and a first allocating module;

[0155] The first determining module is used to determine the first access status information of the target partition corresponding to the partition service based on the first query data through the partition service; the first access status information is used to characterize the size of the read and write load of the target partition.

[0156] The first allocation module is used to add or reduce the number of proxy services in the target partition if the third number of proxy services in the target partition does not match the first access status information.

[0157] In some embodiments, the storage service client further includes a request processing service, which corresponds to multiple proxy services; the device further includes a receiving module, a second determining module, and a first sending module.

[0158] The receiving module is used to receive the first data saving request sent by the client through the request processing service;

[0159] The second determining module is used to determine the target proxy service with the smallest delay by using the request processing service and based on the delay information of each corresponding proxy service within a preset time period; the delay information represents the data transmission and reception duration of the request processing service, and the data transmission and reception duration is the time between sending the first data saving request and receiving the corresponding first query data.

[0160] The first sending module is used to send the first data saving request to the target proxy service through the request processing service.

[0161] In some embodiments, the apparatus further includes: a third determining module, a second sending module, and a second dispatching module;

[0162] The third determining module is used to determine the second access status information of the request processing service by saving the request based on the first data through the request processing service; the second access status information is used to characterize the size of the send and receive load of the request processing service.

[0163] The second sending module is used to send a proxy service allocation request to the partition service if the fourth number of proxy services corresponding to the request processing service does not match the second access status information.

[0164] The second allocation module is used to add or reduce the proxy service corresponding to the request processing service based on the allocation request of the proxy service through the partition service.

[0165] In some embodiments, the apparatus further includes a fourth determining module and a removal module;

[0166] The fourth determination optical module is used to send an exception message to the partition service if the request processing service determines that any of the corresponding proxy services is accessing abnormally.

[0167] The removal module is used to remove proxy services that have access problems if the partition service determines, based on the exception message, that the proxy service has access problems and meets the removal conditions.

[0168] The data storage device and the data storage method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0169] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0170] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0171] Corresponding to the aforementioned embodiments of data storage methods, this disclosure also provides embodiments of data storage systems. Figure 9 is a schematic diagram of a data storage system according to an exemplary embodiment. As shown in Figure 9, the data storage system includes: a storage service client 301, a first storage pool 302, and a second storage pool 303; wherein, the data read / write rate of the second storage pool 303 is lower than the data read / write rate of the first storage pool 302, and the storage service client 301 is configured with a request processing service 3011, a proxy service 3012, and a storage service 3013.

[0172] Request processing service 3011 is used to send the first data save request sent by the client to the corresponding proxy service 3012; the first data save request includes the target data to be saved.

[0173] The proxy service 3012 is used to cache the target data in the memory of the storage service client 301 and send a data caching request to the storage service 3013; send a second data saving request to the storage service 3013 based on the target data in memory; after the target data is transferred from memory to the second storage pool 303, delete the target data in memory and send a data deletion request to the storage service 3013.

[0174] Storage service 3013 is used to cache target data to a first storage pool 304 according to a data caching request; save target data to a second storage pool 305 according to a second data saving request; and delete target data in the first storage pool 304 according to a data deletion request.

[0175] In some implementations, as shown in FIG10, the system also includes a partition service 3014;

[0176] The proxy service 3012 is also used to obtain first query data of target data cached in the first storage pool 302, and send the first query data to the partition service 3014 via the request processing service 3011; and to obtain second query data of target data stored in the second storage pool 303, and send the second query data to the partition service 3014.

[0177] Partition service 3014 is used to provide clients with query services for target data based on the first query data or the second query data.

[0178] The data storage system and data storage method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0179] It is understandable that the above division of modules is only a logical functional division. In actual implementation, the functions of the above modules can be integrated into hardware entities. For example, the functions of the above cache module 201, transfer module 202 and deletion module 203 can be integrated into the processor, etc.

[0180] Based on this, some embodiments of this disclosure also provide an electronic device corresponding to the data storage method provided in the foregoing embodiments. This electronic device can be configured as a storage service terminal as shown in FIG2 to execute the above-described data storage method.

[0181] Figure 11 is a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes: a communication interface 401, a processor 402, a memory 403, and a bus 404; wherein the communication interface 401, the processor 402, and the memory 403 communicate with each other through the bus 404. The processor 402 can execute the data storage method described above by reading and executing machine-executable instructions corresponding to the control logic of the data storage method in the memory 403. The specific content of the method is described in the above embodiment and will not be repeated here.

[0182] The memory 403 mentioned in this embodiment can be any electronic, magnetic, optical, or other physical storage device, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 403 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof. Communication between the system network element and at least one other network element is achieved through at least one communication interface 401 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0183] Bus 404 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 403 is used to store programs, and the processor 402 executes the programs after receiving execution instructions.

[0184] Processor 402 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 402 or by instructions in software form. The processor 402 can be a general-purpose processor, including a network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware controls, etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.

[0185] The electronic device and the data storage method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0186] This disclosure also provides a computer-readable storage medium corresponding to the data storage method provided in the foregoing embodiments. Please refer to FIG12, which shows a computer-readable storage medium 50, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the data storage method provided in any of the foregoing embodiments.

[0187] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0188] The computer-readable storage medium provided in the above embodiments of this disclosure and the data storage method provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0189] This disclosure also provides a computer program product corresponding to the data storage method provided in the foregoing embodiments. The computer program product includes a computer program that is executed by a processor to implement the data storage method provided in the foregoing embodiments.

[0190] The computer program products provided in the above embodiments of this disclosure and the data storage methods provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data storage method, wherein, Applied to the storage service side, the method includes: In response to a first data save request sent by the client, the target data to be saved in the first data save request is cached in the target memory and the first storage pool, respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool; The target data in the target memory is transferred to the second storage pool of the storage system; the data read / write rate of the second storage pool is lower than the data read / write rate of the first storage pool. Delete the target memory and the target data in the first storage pool.

2. The method according to claim 1, wherein, Cache the target data to be saved in the first data save request to the first storage pool, including: The target data is subjected to a first encoding process to obtain first encoded data; the first encoded data includes a first number of first encoded blocks; The first encoded data is cached in the first storage pool.

3. The method according to claim 1 or 2, wherein, The step of transferring the target data in the target memory to the second storage pool of the distributed storage system includes: Read the target data from the target memory; The target data is subjected to a second encoding process to obtain second encoded data; the second encoded data includes a second number of second encoded blocks; The second encoded data is saved to the second storage pool.

4. The method according to claim 1 or 2, wherein, The storage service provider includes a proxy service; The step of caching the target data to be saved in the first data save request to the target memory and the first storage pool respectively includes: caching the target data to the target memory and the first storage pool through the proxy service; The step of transferring the target data in the target memory to the second storage pool of the storage system includes: transferring the target data in the target memory to the second storage pool through the proxy service.

5. The method according to claim 4, wherein, The storage service client also includes a partitioning service, and the method further includes: The proxy service obtains first query data and second query data, and sends the first query data and second query data to the partition service; the first query data is used to query the target data stored in the first storage pool, and the second query data is used to query the target data stored in the second storage pool. The partitioning service provides the client with a query service for the target data based on the first query data or the second query data.

6. The method according to claim 5, wherein, The first query data includes a mapping relationship between the block identifier of the first data block of the target data and the storage location of the first data block in the first storage pool; the second query data includes a mapping relationship between the block identifier and the storage location of the first data block in the second storage pool. The provision of the target data query service to the client based on the first query data or the second query data includes: If the transfer of the target data to the second storage pool is not completed, and a first data query request is received from the client, then according to the target block identifier in the first data query request, the corresponding first storage location is queried in the first query data, and the target data block corresponding to the target block identifier is queried from the first storage location in the first storage pool. If the transfer of the target data to the second storage pool has been completed, and the first data query request is received, then according to the target block identifier, the corresponding second storage location is queried in the second query data, and the target data block corresponding to the target block identifier is queried from the second storage location in the second storage pool.

7. The method according to claim 5 or 6, wherein, The method further includes: The partition service determines the first access status information of the target partition corresponding to the partition service based on the first query data; the first access status information is used to characterize the read and write load of the target partition. If the third number of proxy services in the target partition does not match the first access status information, then the proxy services in the target partition are either added or reduced.

8. The method according to any one of claims 5-7, wherein, The storage service client also includes a request processing service, which corresponds to multiple proxy services. The request processing service receives the first data saving request sent by the client. The request processing service determines the target proxy service with the lowest latency based on the latency information of each proxy service within a preset time period. The delay information represents the data transmission and reception duration of the request processing service, and the data transmission and reception duration is the time between sending the first data saving request and receiving the corresponding first query data. The request processing service sends the first data saving request to the target proxy service.

9. The method according to claim 8, wherein, The method further includes: The request processing service saves the request based on the first data and determines the second access status information of the request processing service; the second access status information is used to characterize the size of the send and receive load of the request processing service. If the fourth number of proxy services corresponding to the request processing service does not match the second access status information, a proxy service allocation request is sent to the partition service. The partitioning service, based on the proxy service allocation request, performs addition or reduction processing on the proxy service corresponding to the request processing service.

10. The method according to claim 8 or 9, wherein, The method further includes: If the request processing service determines that any of the corresponding proxy services is accessing abnormally, an abnormality message is sent to the partition service. If the partition service determines, based on the error message, that the proxy service experiencing access problems meets the removal criteria, then the proxy service experiencing access problems will be removed.

11. A data storage system, wherein, The system includes: a storage service client, a first storage pool, and a second storage pool; the data read / write rate of the second storage pool is lower than that of the first storage pool; the storage service client is configured with a request processing service, a proxy service, and a storage service. The request processing service is used to send the first data save request sent by the client to the corresponding proxy service; the first data save request includes the target data to be saved; The proxy service is used to cache the target data in the memory of the storage service and send a data caching request to the storage service; send a second data saving request to the storage service based on the target data in the memory; and after the target data is transferred from the memory to the second storage pool, delete the target data in the memory and send a data deletion request to the storage service. The storage service is configured to cache the target data in the first storage pool according to the data caching request; save the target data in the second storage pool according to the second data saving request; and delete the target data in the first storage pool according to the data deletion request.

12. The system according to claim 11, wherein, The storage service client is also configured with a partitioning service; The proxy service is further configured to obtain first query data of the target data cached in the first storage pool, and send the first query data to the partition service via the request processing service; and to obtain second query data of the target data stored in the second storage pool, and send the second query data to the partition service; The partitioning service is used to provide the client with a query service for the target data based on the first query data or the second query data.

13. A data storage device, wherein, The device, used in storage services, includes: The caching module is used to respond to a first data save request sent by the client and cache the target data to be saved in the first data save request to the target memory and the first storage pool, respectively; the target memory is the memory of the storage service client, and the storage system where the storage service client is located includes the first storage pool; The transfer module is used to transfer the target data in the target memory to the second storage pool of the storage system; the data read / write rate of the second storage pool is lower than that of the first storage pool. The deletion module is used to delete the target data in the target memory and the first storage pool.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the program to implement the method as described in any one of claims 1-10.

15. A computer-readable storage medium having a computer program stored thereon, wherein, The program is executed by a processor to implement the method as described in any one of claims 1-10.

16. A computer program product comprising a computer program, wherein, The computer program is executed by a processor to implement the method of any one of claims 1-10.

Images