Data storage method, cloud storage system and storage medium

Abstract

The invention discloses a data storage method, a cloud storage system and a storage medium, and the data storage method comprises the steps: receiving to-be-stored data, and storing the to-be-stored data to a cache layer; obtaining the data size of each piece of to-be-stored data in the cache layer, determining the to-be-stored data of which the data size is lower than a preset data threshold in the cache layer to obtain small data, and determining the to-be-stored data of which the data size is not lower than the preset data threshold in the cache layer to obtain big data; aggregating the plurality of small data in the cache layer to obtain a small data aggregation file; and storing the small data aggregation file to an acceleration layer, and storing the big data to a storage layer. The read-write number of the acceleration layer can be reduced, the service life of the storage medium is prolonged, the overall read-write efficiency is improved, the characteristics of different storage media are fully utilized for classified storage of data, and storage resources are reasonably utilized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data storage method, a cloud storage system and a storage medium. BACKGROUND

[0002] With the development of big data technology, data is growing at an extremely fast speed, such as audio and video data, enterprise backup data, business operation log data, etc. Generally, a mechanical hard disk (HDD) or a solid state disk (SSD) is used to store these data.

[0003] However, the storage medium in the current storage system has problems such as short service life and low read-write efficiency. SUMMARY

[0004] To solve the above technical problems, the present application at least provides a data storage method, a cloud storage system and a storage medium.

[0005] The first aspect of the present application provides a data storage method, a storage end including a cache layer, an acceleration layer and a storage layer, the cache layer is used for data caching, the acceleration layer and the storage layer are both used for data storage, the data read-write speed of the acceleration layer is greater than that of the storage layer, the method comprising: receiving to-be-stored data, storing the to-be-stored data to the cache layer; obtaining the data size of each to-be-stored data in the cache layer, determining the to-be-stored data with a data size lower than a preset data threshold in the cache layer, obtaining small data, determining the to-be-stored data with a data size not lower than the preset data threshold in the cache layer, obtaining large data; aggregating multiple small data in the cache layer to obtain a small data aggregation file; storing the small data aggregation file to the acceleration layer, and storing the large data to the storage layer.

[0006] In an embodiment, aggregating multiple small data in the cache layer to obtain a small data aggregation file comprises: obtaining the size of a basic storage unit of the acceleration layer; aggregating multiple small data in the cache layer according to the size of the basic storage unit to obtain a small data aggregation file; wherein the file size of the small data aggregation file does not exceed the size of the basic storage unit.

[0007] In an embodiment, the acceleration layer is composed of a solid state disk; aggregating multiple small data in the cache layer according to the size of the basic storage unit to obtain a small data aggregation file comprises: obtaining the block size and page size of the solid state disk; taking the block size as an upper limit value, and taking the difference between the block size and the page size as a lower limit value; aggregating multiple small data in the cache layer according to the upper limit value and the lower limit value to obtain a small data aggregation file; wherein the file size of the small data aggregation file does not exceed the upper limit value and is not lower than the lower limit value.

[0008] In an embodiment, the method for storing small data aggregation files to the acceleration layer and storing large data to the storage layer comprises: detecting whether the current time is in a peak period or in a flat period; wherein the amount of data to be stored received in the peak period is greater than the amount of data to be stored received in the flat period; if the current time is in the peak period, storing all small data aggregation files to the acceleration layer and storing all large data to the storage layer; if the current time is in the flat period, aggregating multiple large data in the cache layer to obtain large data aggregation files, storing part of the small data aggregation files to the acceleration layer, and storing another part of the small data aggregation files and all the large data aggregation files to the storage layer.

[0009] In an embodiment, the detecting whether the current time is in a peak period or in a flat period comprises: obtaining a pre-set peak period time period and a flat period time period, or collecting load information of the storage end at each time in an initial time period, and setting the peak period time period and the flat period time period based on the load information; detecting whether the current time is in the peak period time period or in the flat period time period to obtain whether the current time is in the peak period or in the flat period.

[0010] In an embodiment, the load information comprises system load, and / or network load, and / or read-write load; the collecting load information of the storage end at each time in an initial time period, and setting the peak period time period and the flat period time period based on the load information comprises: collecting system load, and / or network load, and / or read-write load of the storage end at each time in an initial time period; setting a system peak period time window based on the system load at each time, and / or setting a network peak period time window based on the network load at each time, and / or setting a read-write peak period time window based on the read-write load at each time; fusing the system peak period time window, and / or the network peak period time window, and / or the read-write peak period time window to obtain a comprehensive peak time window; obtaining the peak period time period based on the comprehensive peak time window, and obtaining the flat period time period based on other time periods other than the peak period time period.

[0011] In an embodiment, the method further comprises: obtaining a storage resource usage rate of the acceleration layer; and if the storage resource usage rate exceeds a pre-set usage rate threshold, moving data in the acceleration layer to the storage layer.

[0012] In an embodiment, if the storage resource usage rate exceeds a pre-set usage rate threshold, moving data in the acceleration layer to the storage layer comprises: detecting whether the current time is in a peak period or in a flat period; if the current time is in the peak period, waiting for the end of the peak period, and moving data in the acceleration layer to the storage layer after the end of the peak period; if the current time is in the flat period, directly moving data in the acceleration layer to the storage layer.

[0013] The second aspect of the application provides a data storage device, the device comprising: a data cache module, configured to receive to-be-stored data and store the to-be-stored data to a cache layer; a data division module, configured to obtain data sizes of each to-be-stored data in the cache layer, determine to-be-stored data with a data size lower than a preset data threshold in the cache layer to obtain small data, and determine to-be-stored data with a data size not lower than the preset data threshold in the cache layer to obtain large data; an aggregation module, configured to aggregate a plurality of small data in the cache layer to obtain a small data aggregation file; and a storage module, configured to store the small data aggregation file to an acceleration layer and store the large data to a storage layer.

[0014] The third aspect of the application provides a cloud storage system comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the data storage method.

[0015] The fourth aspect of the application provides a computer-readable storage medium having program instructions stored thereon, the program instructions being executed by a processor to implement the data storage method.

[0016] The above scheme, by receiving to-be-stored data, storing the to-be-stored data to a cache layer, obtaining data sizes of each to-be-stored data in the cache layer, determining to-be-stored data with a data size lower than a preset data threshold in the cache layer to obtain small data, and determining to-be-stored data with a data size not lower than the preset data threshold in the cache layer to obtain large data, aggregating a plurality of small data in the cache layer to obtain a small data aggregation file, and storing the small data aggregation file to an acceleration layer and storing the large data to a storage layer, the number of read and write operations of the acceleration layer can be reduced, the service life of the storage medium can be improved, the overall read and write efficiency can be improved, and the characteristics of different storage media are fully utilized to classify and store data, and storage resources are reasonably utilized.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are incorporated into the specification and form a part of the specification, which illustrate embodiments consistent with the application, and together with the specification, serve to explain the technical solutions of the application.

[0019] Figure 1 is a schematic diagram of a scheme implementation environment illustrated by an exemplary embodiment of the application;

[0020] Figure 2 is a schematic diagram of a storage end illustrated by an exemplary embodiment of the application;

[0021] Figure 3 is a flowchart of a data storage method illustrated by an exemplary embodiment of the application;

[0022] Figure 4 is a schematic diagram of a storage structure according to an example embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a NAND flash according to an example embodiment of the present application;

[0024] Figure 6 is a schematic diagram of dividing peak and flat according to an example embodiment of the present application;

[0025] Figure 7 is a schematic diagram of data storage according to an example embodiment of the present application;

[0026] Figure 8 is a schematic diagram of data migration according to an example embodiment of the present application;

[0027] Figure 9 is a block diagram of a data storage device according to an example embodiment of the present application;

[0028] Figure 10 is a schematic diagram of a cloud storage system according to an example embodiment of the present application;

[0029] Figure 11 is a schematic diagram of a computer readable storage medium according to an example embodiment of the present application. DETAILED DESCRIPTION

[0030] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0031] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, interfaces, techniques, in order to provide a thorough understanding of the present application.

[0032] The term "and / or" herein is merely descriptive of the associated objects, and means that there can be three relationships, for example, A and / or B, which means that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally means that the associated objects are in an "or" relationship. In addition, "multiple" herein means two or more. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C means including any one or more elements selected from the set consisting of A, B and C.

[0033] The inventors of this application have discovered that numerous data read / write operations performed on storage media can affect the lifespan of the storage media. For example, in SSDs, video capture scenarios require continuous writing of numerous images, leading to frequent erase / write operations. This rapidly depletes the program / erase (P / E) cycles of the NAND (Non-volatile Memory Device) flash memory in the SSD, increasing the write amplification factor (WA) and thus affecting the SSD's lifespan. Here, the program / erase cycle refers to the number of times a single flash memory block can be erased and written, and the write amplification factor is the ratio between the actual amount of data written to the SSD and the amount of data written by the user. The smaller the write amplification factor, the longer the SSD's lifespan.

[0034] In addition, storing data of different sizes together can affect the read and write efficiency of the storage medium. For example, for SSDs, images captured in video capture scenarios are written to the SSD, but large images (such as high-definition images larger than 32MB) occupy SSD space but are rarely read repeatedly, resulting in a waste of storage resources and reduced read and write efficiency.

[0035] Furthermore, the fluctuation characteristics of data read and write requests vary at different times. If the same strategy is used for data read and write, it will further reduce the lifespan of the storage medium and the read and write efficiency. For example, the random small data writes during the daytime peak period are in stark contrast to the sequential reading of large files during nighttime analysis.

[0036] To address the aforementioned technical problems, this application provides at least one data storage method, a cloud storage system, and a storage medium.

[0037] The data storage method provided in the embodiments of this application will be described below.

[0038] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of the scheme according to an exemplary embodiment of this application. The implementation environment may include a data source 110 and a storage device 120, which are interconnected.

[0039] The data source 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0040] Storage terminal 120 is equipped with storage resources that can store data. For example, storage terminal 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] Data source 110 sends a data write request to storage 120. Storage 120, upon receiving the data write request, uses storage resources to store the data indicated by the data write request. Alternatively, data source 110 sends a data read request to storage 120. Storage 120, upon receiving the data read request, reads the data indicated by the data write request from storage resources and feeds back the read data to data source 110 or other terminals.

[0042] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the storage end shown in an exemplary embodiment of this application, as follows: Figure 2 As shown, the storage end 120 is implemented using a cloud storage cluster. The cloud storage cluster contains storage nodes, including DN1, DN2, ..., DNn. The data source end 110 writes data to the storage nodes through the SDK (Software Development Kit Client). Each storage node consists of a cache layer, an acceleration layer, and a storage layer. When writing data, it is first stored in the cache layer, and then written to the acceleration layer or storage layer by the cache layer.

[0043] Of course, besides Figure 2 Besides the shown implementation of storage terminal 120, other methods can also be used to implement storage terminal 120, and this application does not limit this.

[0044] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data storage method in an exemplary embodiment of this application. This data storage method can be applied to… Figure 1 The implementation environment is shown, and the execution is carried out by the storage end in the implementation environment.

[0045] like Figure 3 As shown, the data storage method includes at least steps S310 to S340, which are described in detail below:

[0046] Step S310: Receive the data to be stored and store it in the cache layer.

[0047] The storage layer of this application includes a caching layer, an acceleration layer, and a storage layer. The caching layer is used for data caching, while the acceleration layer and the storage layer are both used for data storage.

[0048] The cache layer typically uses volatile storage media for temporary data storage that supports high-speed random access. For example, the cache layer consists of memory, which can be implemented using Dynamic Random Access Memory (DRAM) and / or Static Random Access Memory (SRAM).

[0049] The acceleration layer offers faster data read / write speeds than the storage layer and is used for random data access. The acceleration layer can consist of solid-state drives (SSDs). For example, it can be composed of one or more SSDs, virtualized to create a new storage resource layer. This new storage resource layer serves as the acceleration layer, providing faster data storage compared to the storage layer. Alternatively, the acceleration layer can also be composed of other storage media with similar functionality to SSDs, such as network-attached storage (NAS) and / or external hard drives.

[0050] The storage layer can consist of mechanical hard disk drives (HDDs). Of course, the storage layer can also consist of other storage media with similar functions to mechanical hard disk drives, such as magnetic tapes, optical discs, or hybrid storage devices.

[0051] It should be noted that the specific storage media types used in the caching layer, acceleration layer, and storage layer can be flexibly selected according to the actual application situation, and this application does not impose any restrictions on this.

[0052] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a storage structure as an exemplary embodiment of this application, such as... Figure 4 As shown, the storage end contains multiple storage nodes. Each storage node contains a cache layer, an acceleration layer, and a storage layer. The cache layer is memory, the storage layer consists of hard disk drives (HDDs) (marked with a green box), and the acceleration layer consists of one or more solid-state drives (SSDs) (marked with a red box). Through virtualization, the storage resources of the SSDs are grouped into a storage resource layer to provide accelerated data storage capabilities, such as... Figure 4 The acceleration layer in a single storage node consists of two SSDs.

[0053] Receive the data to be stored and store it in the cache layer.

[0054] The data to be stored refers to the data that needs to be stored, which may be image data, and / or audio data, and / or text data, and / or tabular data, etc. This application does not limit the specific data type of the data to be stored.

[0055] Step S320: Obtain the data size of each data to be stored in the cache layer, determine the data to be stored in the cache layer whose data size is lower than the preset data threshold, and obtain small data; determine the data to be stored in the cache layer whose data size is not lower than the preset data threshold, and obtain large data.

[0056] The size of the data to be stored is used to determine the standard data volume. The larger the size of the data to be stored, the more storage resources are required to store it; the smaller the size of the data to be stored, the less storage resources are required.

[0057] Based on the size of the data to be stored, data smaller than a preset data threshold is classified as small data, and data larger than or equal to the preset data threshold is classified as big data.

[0058] For example, in a video capture scenario, if the data to be stored is images, images smaller than 1MB are classified as small data, and images larger than or equal to 1MB are classified as large data.

[0059] Step S330: In the cache layer, aggregate multiple small data sets to obtain a small data aggregation file.

[0060] Small data sets are aggregated to treat multiple small data sets as a whole, resulting in a small data aggregation file.

[0061] The size of the aggregated small data file can be a pre-set empirical value or a value flexibly calculated based on the actual situation of the storage medium in the acceleration layer; there is no limitation on this.

[0062] Step S340: Store small data aggregation files to the acceleration layer and large data to the storage layer.

[0063] The caching layer transfers small data aggregation files to the acceleration layer for storage, and transfers large data files to the storage layer for storage.

[0064] The data storage method provided in this application reduces the number of reads and writes in the acceleration layer and extends the lifespan of the storage medium by aggregating small data and storing it in the storage layer. It also makes full use of the characteristics of different storage media to classify and store data, storing small data that is more likely to be read repeatedly in the acceleration layer with faster read and write speeds and storing large data that is less likely to be read repeatedly in the storage layer, thereby improving overall read and write efficiency and making reasonable use of storage resources.

[0065] Next, some embodiments of this application will be described in detail.

[0066] In step S330, multiple small data sets are aggregated in the cache layer to obtain a small data aggregation file, including the following steps S331 to S332.

[0067] Step S331: Obtain the size of the basic storage unit of the acceleration layer.

[0068] Different types of storage media have different basic storage units. For example, the basic storage unit of an SSD is a block, which is composed of multiple pages.

[0069] Step S332: According to the size of the basic storage unit, aggregate multiple small data in the cache layer to obtain a small data aggregation file; wherein, the file size of the small data aggregation file does not exceed the size of the basic storage unit.

[0070] Based on the size of the basic storage unit, the size of the small data aggregation file is limited. Multiple small data are aggregated in the cache layer to obtain a small data aggregation file whose file size does not exceed the size of the basic storage unit. The final aggregated small data file is then sent to the acceleration layer for storage.

[0071] In some implementations, the acceleration layer consists of a solid-state drive (SSD); in step S332, multiple small data are aggregated in the cache layer according to the size of the basic storage unit to obtain a small data aggregation file, including the following steps S3321 to S3323.

[0072] Step S3321: Obtain the block size and page size of the solid-state drive.

[0073] Step S3322: Use the block size as the upper limit and the difference between the block size and the page size as the lower limit.

[0074] Step S3323: According to the upper limit and lower limit, aggregate multiple small data in the cache layer to obtain a small data aggregation file; wherein, the file size of the small data aggregation file does not exceed the upper limit and is not lower than the lower limit.

[0075] SSDs are primarily composed of NAND flash memory chips. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of NAND flash memory illustrated in an exemplary embodiment of this application, as shown below. Figure 5As shown, the basic units for reading and writing NAND flash memory are blocks and pages. A block consists of multiple pages. NAND flash memory data reading and writing operations mainly involve erasing, programming, and reading. The basic unit for programming and reading is a page, while the basic unit for erasing is a block. Before writing data to a page, the entire block containing that page must be erased. Therefore, when writing data to a page, other valid data in the block needs to be copied to a new storage space, thereby erasing the original entire block. This process is called garbage collection (GC). Each programming write and erasure operation on a block is called a P / E (Program / Erase) cycle.

[0076] There is a limit to the number of times each block can be erased and written in an SSD. If some blocks are erased and written too many times, they will become unusable.

[0077] This application improves the lifespan of SSDs by first aggregating small data in the cache layer and then storing the aggregated small data files to the SSD.

[0078] For example, if we define the size of an SSD page as "page" and the size of an SSD block as "block", then we can set the upper limit of the small data aggregation file to "block" and the lower limit to "block-page". Therefore, the file size range of the small data aggregation file is "block-page". <Size<=block。

[0079] In the cache layer, multiple small data points are aggregated. If the total size of the currently aggregated small data points is within the upper limit and not lower than the lower limit, the aggregation ends, and the final small data aggregation file is obtained.

[0080] In the above embodiments, the size of the aggregated file is limited based on the size of the basic storage unit of the acceleration layer. The basic storage unit is used as the storage area for the entire small data aggregated file, which improves the utilization of storage resources, improves read and write efficiency through batch operations, and avoids frequent erasure and writing of storage media such as SSDs, thereby improving the lifespan of storage resources.

[0081] Of course, besides limiting the file size of the small data aggregation file in the above embodiments, other methods can also be used to limit the file size of the small data aggregation file. For example, the file size of the small data aggregation file can be preset based on experience. Alternatively, other methods can be used to generate the final small data aggregation file, such as aggregating small data whose data collection time is within a preset time period to obtain the final small data aggregation file. This application does not limit the specific aggregation method of the small data aggregation file.

[0082] After aggregating to obtain small data aggregation files, the small data aggregation files are stored in the acceleration layer, and the large data is stored in the storage layer.

[0083] In some implementations, step S340 involves storing small data aggregation files to the acceleration layer and storing large data to the storage layer, including steps S341 to S344 below.

[0084] Step S341: Detect whether the current period is a peak period or a non-peak period.

[0085] The amount of data to be stored received during peak periods is greater than the amount of data to be stored received during off-peak periods.

[0086] For example, peak and off-peak time periods can be preset based on experience, the preset peak and off-peak time periods can be obtained, the current time can be detected to be in a peak or off-peak time period, and it can be determined whether the current time is in a peak or off-peak period.

[0087] For example, 6-9 am and 4-7 pm can be designated as peak hours, while the rest of the time can be designated as off-peak hours.

[0088] For example, specific operating parameters of the storage device can also be detected, and the current peak or off-peak period can be determined based on the operating parameters.

[0089] For example, step S341 detects whether the current period is a peak period or a non-peak period, including the following steps S3411 to S3412.

[0090] Step S3411: Collect the load information of the storage terminal at various times during the initial time period, and set the peak period and off-peak period based on the load information.

[0091] Step S3412: Detect whether the current time is in a peak or off-peak period, and determine whether the current time is in a peak or off-peak period.

[0092] For example, load information consists of one or more load metrics, specifically including system load, and / or network load, and / or read / write load. The system load, and / or network load, and / or read / write load of the storage terminal at various times within an initial time period are collected. A system peak time window is set based on the system load at each time point, and / or a network peak time window is set based on the network load at each time point, and / or a read / write peak time window is set based on the read / write load at each time point. The system peak time window, and / or the network peak time window, and / or the read / write peak time window are merged to obtain a comprehensive peak time window. The peak time period is obtained based on the comprehensive peak time window, and the off-peak time period is obtained based on other time periods outside the peak time period.

[0093] Among them, system load is used to characterize the busyness of storage system resources, such as the busyness of the central processing unit (CPU), network load is used to characterize the busyness of storage network bandwidth resources, and read / write load is used to characterize the busyness of data read requests and data write requests processed by the storage unit per unit time.

[0094] To illustrate this with a specific application scenario, the entire time period is initialized as off-peak hours. Load data recording is initiated, and the system load (AvgLoad), network load (NetLoad), and read / write load (IOLoad) of the storage terminal are collected at preset intervals (e.g., 10 seconds). Furthermore, the corresponding thresholds for each of these load metrics are calculated. For example, the threshold for system load (TAvgLoad) is equal to twice the number of logical cores of the Central Processing Unit (CPU); the threshold for network load (TNetLoad) is equal to the total bandwidth of the network interface card (NIC); and the threshold for read / write load (TIOLoad) is equal to the ratio of CPU time spent waiting for I / O to total CPU time.

[0095] If the system load AvgLoad is greater than TAvgLoad for a continuous 10 minutes, and / or the network load NetLoad > 70% * TNetLoad, and the read / write load TIOLoad > 30%, then the system switches to peak period, and the corresponding peak period time window is recorded.

[0096] Then, the peak time windows of each load indicator for the day are obtained, and the peak time windows of each load indicator are merged to obtain the final comprehensive peak time window.

[0097] For example, please see Figure 6 , Figure 6 This is a schematic diagram illustrating the division of peak and off-peak periods as shown in an exemplary embodiment of this application, such as... Figure 6As shown, the union of the system peak time window (blue dashed line), the network peak time window (red dashed line), and the read / write peak time window (green solid line) is obtained to get the final comprehensive peak time window.

[0098] Then, the time period corresponding to the comprehensive peak time window is taken as the peak period, and the other time periods outside the peak period are taken as the off-peak period.

[0099] Of course, in addition to using the union method to fuse peak time windows in the above embodiments, other methods can also be used to fuse peak time windows. For example, the intersection of each peak time window can be obtained to get the final comprehensive peak time window.

[0100] In addition to the method described in the above embodiments of obtaining peak time windows corresponding to each load indicator to determine peak time periods, other methods can also be used to determine peak time periods. For example, based on the system load, and / or network load, and / or read / write load at each moment, the probability of a peak period at each moment can be determined. The higher the system load, and / or network load, and / or read / write load, the higher the probability of a peak period at the corresponding moment; conversely, the lower the system load, network load, and read / write load, the lower the probability of a peak period at the corresponding moment. Moments with a peak probability greater than a preset probability threshold are divided into peak time periods, and other time periods outside of peak time periods are obtained as off-peak time periods.

[0101] In another embodiment, the system load, and / or network load, and / or read / write load at the current moment are periodically acquired. If the system load, and / or network load, and / or read / write load are greater than a preset load threshold for a continuous preset time, it is determined that the current period is a peak period; if the system load, and / or network load, and / or read / write load are not greater than the preset load threshold for a continuous preset time, it is determined that the current period is a non-peak period.

[0102] Of course, in addition to the system load, and / or network load, and / or read / write load in the above embodiments, other load parameters can be used to determine peak or off-peak periods, and this application does not limit this.

[0103] By combining load information in the above embodiments, peak and off-peak periods can be flexibly determined, thereby improving the accuracy of peak and off-peak period division.

[0104] Step S342: If it is during peak period, proceed to step S343; if it is during off-peak period, proceed to step S344.

[0105] Step S343: Store all small data aggregation files in the acceleration layer and store all large data in the storage layer.

[0106] Step S344: Aggregate multiple large data in the cache layer to obtain a large data aggregation file, store some small data aggregation files in the acceleration layer, and store another part of the small data aggregation files and all large data aggregation files in the storage layer.

[0107] Further aggregate the large data to obtain a large data aggregation file. Among them, the aggregation method of the large data can be the same as or different from the aggregation method of the small data.

[0108] For example, obtain the size of the basic storage unit of the storage layer; based on the size of the basic storage unit of the storage layer, set the aggregation file size; according to the aggregation file size, aggregate multiple large data in the cache layer to obtain a large data aggregation file; among them, the file size of the large data aggregation file does not exceed the aggregation file size.

[0109] If the storage medium type of the storage layer is different, the basic storage unit of the storage layer is also different.

[0110] For example, assume that the storage layer is HDD, then the basic storage unit is a sector (Sector), please refer to 7, Figure 7 is a schematic diagram of data storage shown in an exemplary embodiment of the present application, as Figure 7 shown, the storage policy includes:

[0111] During the off-peak period, the method of storing small data in the cache layer to the storage layer and the acceleration layer can be: aggregate 1 / 3 of the small data to obtain a small data aggregation file corresponding to the acceleration layer, and judge whether the file size Size1 of the small data aggregation file corresponding to the acceleration layer satisfies: block-page < Size1 <= block. If it is satisfied, store the small data aggregation file in the acceleration layer; aggregate 2 / 3 of the small data to obtain a small data aggregation file corresponding to the storage layer, and judge whether the file size Size2 of the small data aggregation file corresponding to the storage layer satisfies: (n - 1)*Sector < Size2 <= n*Sector. If it is satisfied, store the small data aggregation file in the storage layer, where the value of n can be preset according to experience, such as n = 10; or, judge whether the aggregation duration t1 of the small data aggregation file corresponding to the storage layer is equal to the first preset time threshold T1. If it is equal, store the small data aggregation file in the storage layer.

[0112] During the off-peak period, the method of storing all large data in the cache layer to the storage layer can be: judge whether the file size Size3 of the large data aggregation file satisfies: (n - 1)*Sector < Size3 <= n*Sector. By dividing multiple sectors into an aggregation file storage area, the HDD addressing overhead can be reduced, and the read and write efficiency can be improved through batch operations.

[0113] Alternatively, during the low-traffic period, the method of storing all the big data in the cache layer to the storage layer can also be: determining whether the aggregation duration t2 of the big data aggregation file corresponding to the storage layer is equal to the first preset time threshold T2. If it is equal, then storing the big data aggregation file to the storage layer.

[0114] During the peak period, the method of storing all the small data in the cache layer to the acceleration layer can be: aggregating all the small data to obtain the small data aggregation file corresponding to the acceleration layer, and determining whether the file size Size4 of the small data aggregation file corresponding to the acceleration layer satisfies: block-page < Size4 <= block. If it satisfies, then storing the small data aggregation file to the acceleration layer.

[0115] During the peak period, directly store all the big data in the cache layer to the storage layer.

[0116] In the above embodiments, considering the fluctuating characteristics of data storage at different times, different storage strategies are adopted during the low-traffic period and the peak period. During the low-traffic period, part of the small data and big data are aggregated and stored to the storage layer, which improves the read and write efficiency of the storage layer and can avoid a large amount of data read and write to the acceleration layer during the low-traffic period, reducing the service life of the acceleration layer. During the peak period, directly store all the big data to the storage layer and aggregate the small data and store it to the acceleration layer, which can store the data in a timely manner, ensure the execution of the data storage service, and improve the utilization rate of system resources.

[0117] In some embodiments, the method further includes: obtaining the storage resource utilization rate of the acceleration layer; if the storage resource utilization rate exceeds the preset utilization rate threshold, then moving the data in the acceleration layer to the storage layer.

[0118] Among them, the storage resource utilization rate is used to indicate the quantity of the storage resources that have been used.

[0119] For example, the storage resource utilization rate is the ratio between the quantity of the used storage resources of the acceleration layer and the total quantity of the storage resources.

[0120] Judge whether to move the data in the acceleration layer according to the storage resource utilization rate of the acceleration layer. For example, if the storage resource utilization rate exceeds 30%, then move the data in the acceleration layer to the storage layer.

[0121] Exemplarily, if the storage resource utilization rate exceeds the preset utilization rate threshold, then moving the data in the acceleration layer to the storage layer includes: detecting whether it is currently in the peak period or in the low-traffic period; if it is in the peak period, then wait until the peak period ends, and after the peak period ends, move the data in the acceleration layer to the storage layer; if it is in the low-traffic period, then directly move the data in the acceleration layer to the storage layer.

[0122] For example, please refer to Figure 8 ,Figure 8 This is a schematic diagram illustrating data transfer in an exemplary embodiment of this application, as shown below. Figure 8 As shown, the data migration strategy includes: starting a timer (e.g., 8 hours); if the timer expires, setting the data migration flag Tdata to 1, determining whether the current period is off-peak or peak, obtaining the storage resource utilization rate (Cusage) of the acceleration layer, and stopping the timer; determining whether the storage resource utilization rate (Cusage) meets the condition (Cusage < 30%). If the condition is met, determining whether the current period is peak; if it is peak, waiting for the current peak period to end, and then continuing to determine whether the storage resource utilization rate (Cusage) meets the condition (Cusage < 30%); if it is not peak, continuing to start the timer. If the condition (Cusage < 30%) is not met, determining whether the current period is peak; if it is peak, waiting for the current peak period to end, and then migrating the data from the acceleration layer to the storage layer; if it is not peak, directly migrating the data from the acceleration layer to the storage layer. After the data migration is completed, the data migration flag Tdata is set to 0, and the next timing cycle begins, starting the timer again.

[0123] Optionally, when moving data from the acceleration layer to the storage layer, hot data in the acceleration layer can be identified. For example, data with a total access count greater than 3 can be marked as hot data. Hot data is not moved to ensure that it can be read and written quickly.

[0124] The above embodiments improve resource utilization and ensure the lifespan of the acceleration layer storage media by determining whether to perform data migration based on the available storage resources of the acceleration layer. For example, after a long period of writing, the usable capacity of an SSD will decrease. In this case, writing data will cause the write amplification factor to increase rapidly. Timely data migration can effectively reuse the SSD and avoid the write amplification factor from increasing rapidly.

[0125] Figure 9 This is a block diagram illustrating a data storage device according to an exemplary embodiment of this application. Figure 9 As shown, the exemplary data storage device 900 includes:

[0126] Data caching module 910 is used to receive data to be stored and store the data to be stored in the cache layer;

[0127] The data partitioning module 920 is used to obtain the data size of each data to be stored in the cache layer, determine the data to be stored in the cache layer whose data size is lower than the preset data threshold, and obtain small data; and determine the data to be stored in the cache layer whose data size is not lower than the preset data threshold, and obtain big data.

[0128] The aggregation module 930 is used to aggregate multiple small data sets in the cache layer to obtain a small data aggregation file;

[0129] Storage module 940 is used to store small data aggregation files to the acceleration layer and large data to the storage layer.

[0130] It should be noted that the data storage device provided in the above embodiments and the data storage method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data storage device provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0131] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the structure of a cloud storage system according to an exemplary embodiment of this application. The cloud storage system 1000 includes a memory 1001 and a processor 1002. The processor 1002 is used to execute program instructions stored in the memory 1001 to implement the steps in any of the above-described data storage method embodiments. In a specific implementation scenario, the cloud storage system 1000 is implemented based on devices such as microcomputers, servers, and mobile devices, and is not limited thereto.

[0132] Specifically, processor 1002 controls itself and memory 1001 to implement the steps in any of the above data storage method embodiments. Processor 1002 may also be referred to as a Central Processing Unit (CPU). Processor 1002 may be an integrated circuit chip with signal processing capabilities. Processor 1002 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 1002 may be implemented using integrated circuit chips.

[0133] Please see Figure 11 , Figure 11This is a schematic diagram illustrating the structure of a computer-readable storage medium according to an exemplary embodiment of this application. The computer-readable storage medium 1100 stores program instructions 1110 that can be executed by a processor. The program instructions 1110 are used to implement the steps in any of the above-described data storage method embodiments.

[0134] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0135] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data storage method, characterized by, The storage end comprises a cache layer, an acceleration layer and a storage layer, the cache layer is used for data caching, the acceleration layer and the storage layer are used for data storage, the data read-write speed of the acceleration layer is greater than that of the storage layer, and the method comprises: receiving to-be-stored data, storing the to-be-stored data into the cache layer; obtaining the data size of each to-be-stored data in the cache layer, determining to-be-stored data with a data size lower than a preset data threshold in the cache layer, obtaining small data, and determining to-be-stored data with a data size not lower than the preset data threshold in the cache layer, obtaining large data; aggregating multiple small data in the cache layer to obtain a small data aggregation file; storing the small data aggregation file into the acceleration layer and storing the large data into the storage layer.

2. The method of claim 1, wherein, The aggregation of multiple small data in the cache layer to obtain a small data aggregation file comprises: obtaining the size of a basic storage unit of the acceleration layer; aggregating multiple small data in the cache layer according to the size of the basic storage unit to obtain a small data aggregation file, wherein the file size of the small data aggregation file does not exceed the size of the basic storage unit.

3. The method of claim 2, wherein, The acceleration layer is composed of a solid state disk; the aggregation of multiple small data in the cache layer according to the size of the basic storage unit to obtain a small data aggregation file comprises: obtaining the block size and page size of the solid state disk; taking the block size as an upper limit value, and taking the difference between the block size and the page size as a lower limit value; aggregating multiple small data in the cache layer according to the upper limit value and the lower limit value to obtain a small data aggregation file, wherein the file size of the small data aggregation file does not exceed the upper limit value and is not lower than the lower limit value.

4. The method of claim 1, wherein, The storage of the small data aggregation file into the acceleration layer and the storage of the large data into the storage layer comprises: detecting whether the current time is in a peak period or a flat peak period; wherein the number of to-be-stored data received in the peak period is greater than that in the flat peak period; if the current time is in the peak period, storing all the small data aggregation files into the acceleration layer and storing all the large data into the storage layer; if the current time is in the flat peak period, aggregating multiple large data in the cache layer to obtain a large data aggregation file, storing part of the small data aggregation files into the acceleration layer, and storing another part of the small data aggregation files and all the large data aggregation files into the storage layer.

5. The method of claim 4, wherein, The detection of whether the current time is in the peak period or the flat peak period comprises: obtaining a preset peak period time period and a flat peak period time period, or collecting load information of the storage end at each time in an initial time period, and setting a peak period time period and a flat peak period time period based on the load information; detecting whether the current time is in the peak period time period or the flat peak period time period to obtain whether the current time is in the peak period or the flat peak period.

6. The method of claim 5, wherein, The load information includes system load, and / or network load, and / or read-write load; the load information of the storage end at each time point in the initial time period is collected, the peak time period and the flat peak time period are set based on the load information, and the method comprises: collecting system load, and / or network load, and / or read-write load of the storage end at each time point in the initial time period; setting a system peak time window based on the system load at each time point, and / or setting a network peak time window based on the network load at each time point, and / or setting a read-write peak time window based on the read-write load at each time point; fusing the system peak time window, and / or the network peak time window, and / or the read-write peak time window to obtain a comprehensive peak time window; obtaining the peak time period based on the comprehensive peak time window, and obtaining the flat peak time period based on other time periods other than the peak time period.

7. The method of claim 1, wherein, The method further comprises: obtaining the storage resource utilization rate of the acceleration layer; if the storage resource utilization rate exceeds the preset utilization threshold, moving the data in the acceleration layer to the storage layer.

8. The method of claim 7, wherein, If the storage resource utilization rate exceeds the preset utilization threshold, the data in the acceleration layer is moved to the storage layer, comprising: detecting whether the current is in the peak period or in the flat peak period; if in the peak period, waiting for the end of the peak period, and moving the data in the acceleration layer to the storage layer after the end of the peak period; if in the flat peak period, directly moving the data in the acceleration layer to the storage layer.

9. A cloud storage system, characterized by, The cloud storage system comprises a memory and a processor, and the processor is used to execute program instructions stored in the memory to realize the steps in the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, and the program instructions can be executed by the processor to realize the steps in the method of any one of claims 1-8.

Images