Data storage method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a data storage method and device, electronic equipment, a storage medium and a program product. The storage cost can be effectively reduced. The method comprises the following steps: setting an archiving life cycle for a target storage bucket in a first storage system, wherein the first storage system is a storage system of which storage data can be accessed in real time; under the condition that at least one piece of data meeting a target condition exists in the target storage bucket, the at least one piece of data is transferred from the first storage system to an archiving storage system, and the archiving storage system is a storage system in which stored data cannot be accessed in real time; wherein the target condition comprises at least one of the following items: the access frequency of the data in the archiving life cycle is less than or equal to a frequency threshold; the duration from the access time of the last access of the data in the archiving life cycle to the current moment is greater than or equal to a duration threshold.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data storage, and in particular to a data storage method, device, electronic device, storage medium and program product. Background Art

[0002] As the amount of video data produced by video companies continues to grow, a lot of data stored in cold storage or standard storage has not been accessed for a long time, resulting in a waste of storage costs to a certain extent. As the amount of data continues to grow and accumulate, the scale of wasted costs will become larger and larger. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data storage method, device, electronic device, storage medium and program product.

[0004] According to a first aspect of an embodiment of the present disclosure, a data storage method is provided, the method comprising: setting an archiving lifecycle for a target storage bucket in a first storage system, the first storage system being a storage system in which stored data can be accessed in real time; if there is at least one data satisfying a target condition in the target storage bucket, transferring the at least one data from the first storage system to an archiving storage system, the archiving storage system being a storage system in which stored data cannot be accessed in real time; wherein the target condition comprises at least one of the following: the access frequency of the data within the archiving lifecycle is less than or equal to a frequency threshold; the access time of the last access of the data within the archiving lifecycle is greater than or equal to a duration threshold from the current moment.

[0005] In some embodiments of the present disclosure, setting an archiving lifecycle for a target bucket in a first storage system includes: setting an archiving lifecycle for the target bucket in the first storage system when a bucket identifier of the target bucket is not in a bucket blacklist; wherein the bucket blacklist includes at least one bucket identifier, and the bucket identifier is used to indicate a bucket.

[0006] In some embodiments of the present disclosure, the target condition also includes: the data does not include a data tag in a tag blacklist, the tag blacklist includes at least one data tag, and the data tag is used to indicate that the corresponding data cannot be archived.

[0007] In some embodiments of the present disclosure, the target condition includes that the access frequency of the data during the archiving life cycle is less than or equal to a frequency threshold; the method also includes: determining the access frequency of the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket; and / or, the target condition includes the access time of the last access to the data during the archiving life cycle, the duration from the current moment is greater than or equal to a duration threshold; the method also includes: determining the access time of the last access to the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket.

[0008] In some embodiments of the present disclosure, after transferring the at least one data from the first storage system to the archival storage system, the method further includes: receiving a data access request for target data in the at least one data, the data access request carrying the length of time the thawed state corresponding to the target data is maintained; in response to the data access request, copying the target data from the archival storage system to obtain first data; storing the first data in a second storage system, the second storage system being a storage system in which the stored data can be accessed in real time; and deleting the first data stored in the second storage system after the thawed state is maintained for a certain length of time.

[0009] In some embodiments of the present disclosure, the data access request also carries a thawing priority for thawing the target data; in response to the data access request, the target data is copied from the archive storage system to obtain the first data, including: in response to the data access request, when there are multiple data to be thawed in the archive storage system, according to the thawing priority, determining the thawing order of the target data, the multiple data to be thawed including the target data; according to the thawing order, copying the target data from the archive storage system to obtain the first data.

[0010] In some embodiments of the present disclosure, after receiving a data access request for target data in the at least one data, the method further includes: determining a cumulative number of accesses to the target data within a preset time period; and in response to the data access request, copying the target data from the archive storage system to obtain first data, including: in a case where the cumulative number of accesses is less than a number threshold, in response to the data access request, copying the target data from the archive storage system to obtain the first data.

[0011] In some embodiments of the present disclosure, after determining the cumulative number of accesses to the target data within a preset time period, the method further includes: when the cumulative number of accesses is greater than or equal to a threshold number, in response to the data access request, copying the target data from the archive storage system to obtain second data; deleting the target data stored in the archive storage system; and storing the second data in a third storage system, the third storage system being a storage system in which the stored data can be accessed in real time.

[0012] In some embodiments of the present disclosure, after receiving the data access request for the target data in the at least one data, the method further includes:

[0013] In the case that the target data has not been completely unfrozen, a prompt message is returned in response to the data access request, and the prompt message is used to prompt the user to wait for the target data to be unfrozen.

[0014] In some embodiments of the present disclosure, after returning prompt information in response to the data access request, the method further includes:

[0015] receiving the data access request again;

[0016] In response to the data access request, when the target data is unfrozen, the unfrozen target data is returned.

[0017] According to a second aspect of an embodiment of the present disclosure, a data storage device is provided, the device comprising: a setting module, used to set an archiving lifecycle for a target storage bucket in a first storage system, the first storage system being a storage system in which stored data can be accessed in real time; a transfer module, used to transfer at least one data from the first storage system to an archiving storage system, when at least one data satisfying a target condition exists in the target storage bucket, the archiving storage system being a storage system in which stored data cannot be accessed in real time; wherein the target condition comprises at least one of the following: the access frequency of the data within the archiving lifecycle is less than or equal to a frequency threshold; the access time of the last access of the data within the archiving lifecycle is greater than or equal to a duration threshold from the current moment.

[0018] In some embodiments of the present disclosure, the setting module is specifically used to set an archiving lifecycle for a target bucket in a first storage system when the bucket identifier of the target bucket is not in a bucket blacklist; wherein the bucket blacklist includes at least one bucket identifier, which is used to indicate a bucket.

[0019] In some embodiments of the present disclosure, the target condition also includes: the data does not include a data tag in a tag blacklist, the tag blacklist includes at least one data tag, and the data tag is used to indicate that the corresponding data cannot be archived.

[0020] In some embodiments of the present disclosure, the target condition includes that the access frequency of the data during the archiving life cycle is less than or equal to a frequency threshold; the device also includes: a determination module for determining the access frequency of the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket; and / or, the target condition includes the access time of the last access to the data during the archiving life cycle, the duration from the current moment being greater than or equal to a duration threshold; the device also includes: a determination module for determining the access time of the last access to the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket.

[0021] In some embodiments of the present disclosure, the device also includes: a receiving module, a copying module, a storage module and a deletion module, the receiving module being used to receive a data access request for target data in the at least one data after transferring the at least one data from the first storage system to the archival storage system, the data access request carrying the thaw state retention time corresponding to the target data; the copying module being used to copy the target data from the archival storage system to obtain the first data in response to the data access request; the storage module being used to store the first data in a second storage system, the second storage system being a storage system in which the stored data can be accessed in real time; the deletion module being used to delete the first data stored in the second storage system after the thaw state is maintained for a certain time.

[0022] In some embodiments of the present disclosure, the data access request also carries a thawing priority for thawing the target data; the copy module is specifically used to respond to the data access request and, when there are multiple data to be thawed in the archive storage system, determine the thawing order of the target data according to the thawing priority, and the multiple data to be thawed include the target data; and copy the target data from the archive storage system according to the thawing order to obtain the first data.

[0023] In some embodiments of the present disclosure, the device also includes: a determination module, used to determine the cumulative number of accesses to the target data within a preset time period after receiving a data access request for the target data in the at least one data; and a copy module, specifically used to copy the target data from the archive storage system to obtain the first data in response to the data access request when the cumulative number of accesses is less than a number threshold.

[0024] In some embodiments of the present disclosure, the copy module is further used to, after determining the cumulative number of accesses to the target data within a preset time period, copy the target data from the archive storage system in response to the data access request to obtain the second data when the cumulative number of accesses is greater than or equal to a threshold number; the device also includes: a deletion module, used to delete the target data stored in the archive storage system; the storage module is also used to store the second data in a third storage system, the third storage system being a storage system in which the stored data can be accessed in real time.

[0025] In some embodiments of the present disclosure, the device also includes: a return module, which is used to receive a data access request for target data in the at least one data, and return a prompt message in response to the data access request if the target data has not been unfrozen, wherein the prompt message is used to prompt the user to wait for the target data to be unfrozen.

[0026] In some embodiments of the present disclosure, the receiving module is further used to receive the data access request again after returning prompt information in response to the data access request; the returning module is further used to return the unfrozen target data in response to the data access request when the target data is unfrozen.

[0027] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the data storage method described in the first aspect is implemented.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data storage method according to the first aspect is implemented.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program. When the computer program product runs on a processor, the processor executes the computer program to implement the data storage method as described in the first aspect.

[0030] In a sixth aspect of an embodiment of the present disclosure, a chip is provided, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run program instructions to implement the data storage method as described in the first aspect.

[0031] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: In the embodiment of the present disclosure, an archiving lifecycle is set for a target storage bucket in a first storage system, and the first storage system is a storage system in which stored data can be accessed in real time; when there is at least one data satisfying a target condition in the target storage bucket, the at least one data is transferred from the first storage system to an archiving storage system, and the archiving storage system is a storage system in which stored data cannot be accessed in real time; wherein the target condition includes at least one of the following: the access frequency of the data within the archiving lifecycle is less than or equal to the frequency threshold; the access time of the last access of the data within the archiving lifecycle is greater than or equal to the duration threshold from the current time. In this way, by setting the archiving lifecycle at the bucket granularity in the first storage system, and then transferring the data in the bucket whose access frequency during the archiving lifecycle is less than or equal to the frequency threshold and / or the access time of the last access during the archiving lifecycle is greater than or equal to the duration threshold from the current time, to the archiving storage system for storage, thereby transferring the data with a low access frequency and no access for a long time during the archiving lifecycle to the archiving storage, thereby reducing the waste of storage costs to a certain extent. Moreover, setting the archiving lifecycle at the bucket granularity can improve the setting efficiency of the archiving lifecycle compared to setting the archiving lifecycle for each data, thereby improving the archiving processing efficiency of the data and effectively reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 One of the flowcharts of the data storage method provided in the embodiment of the present disclosure;

[0035] Figure 2 A second flowchart of the data storage method provided by the embodiment of the present disclosure;

[0036] Figure 3 A third flowchart of the data storage method provided by the embodiment of the present disclosure;

[0037] Figure 4 A structural block diagram of a data storage device provided in an embodiment of the present disclosure;

[0038] Figure 5 A structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0041] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0042] First, some nouns or terms involved in the claims and description of the present invention are explained below.

[0043] Cold storage, hot storage, standard storage, and archive storage are common storage types in the data storage field. They each have different characteristics and applicable scenarios:

[0044] Cold storage: Cold storage is used to store data that is rarely accessed or used. Its characteristics include: low storage cost and slow access speed.

[0045] Hot storage: Hot storage is suitable for data that needs to be accessed frequently and quickly. Its characteristics include: high performance, fast read and write speed, able to meet the needs of real-time processing and high concurrent access; relatively high cost because it requires high-performance hardware and technical support.

[0046] Standard storage: Standard storage strikes a balance between performance and cost. Its features include: providing relatively stable read and write performance and relatively moderate cost.

[0047] Archive storage: Archive storage is mainly used for long-term storage of data with very low access frequency. Its characteristics include: high data reliability and security; relatively low storage cost.

[0048] The electronic device in the embodiments of the present disclosure may be a mobile electronic device or a non-mobile electronic device. The mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA); the non-mobile electronic device may be a personal computer (PC), a television (TV), a teller machine, or an automated machine; the embodiments of the present disclosure do not specifically limit this.

[0049] The executor of the data storage method provided in the embodiments of the present disclosure may be the above-mentioned electronic device (including mobile electronic devices and non-mobile electronic devices), or may be a functional module and / or functional entity in the electronic device that can implement the data storage method. The specific execution subject may be determined based on actual usage requirements, and the embodiments of the present disclosure are not limited thereto.

[0050] The data storage method provided by the embodiment of the present disclosure is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown, an embodiment of the present disclosure provides a data storage method, which may include the following steps 101 to 102.

[0052] 101. Set an archiving lifecycle for a target storage bucket in a first storage system.

[0053] The archiving life cycle can be set according to actual use requirements and is not limited here. For example, the archiving life cycle can be 180 days or 360 days.

[0054] The archive life cycle is not a fixed period of time, but a time window before the scan time at each scan, with a duration equal to the duration of the archive life cycle.

[0055] Different storage buckets may be set with different archiving life cycles, or may be set with the same archiving life cycle, which is not limited here.

[0056] The first storage system is a storage system in which stored data can be accessed in real time, and the first storage system includes any one of the following: a hot storage system, a cold storage system, and a standard storage system.

[0057] Typically, the first storage system includes a cold storage system or a standard storage system.

[0058] 102. If at least one data satisfying the target condition exists in the target storage bucket, transfer the at least one data from the first storage system to the archival storage system.

[0059] Among them, the archiving storage system is a storage system in which the stored data cannot be accessed in real time.

[0060] Among them, the target condition includes at least one of the following: the access frequency of the data within the archive life cycle is less than or equal to the frequency threshold; the access time of the last access to the data within the archive life cycle is greater than or equal to the duration threshold from the current time.

[0061] The frequency threshold and duration threshold can be determined according to actual conditions and are not limited here. For example, the frequency threshold can be 0, 1, or 2, etc.; the duration threshold can be 15 days, 30 days, or 90 days, etc.

[0062] The duration threshold is smaller than the duration of the archive life cycle.

[0063] It can be understood that the access frequency of the data during the archiving life cycle is less than or equal to the frequency threshold, which means that the access frequency of the data during the archiving life cycle is very low and can be archived to reduce storage costs; the access time of the last access to the data during the archiving life cycle is greater than or equal to the duration threshold from the current moment, which means that the data has not been accessed for a long time before the current moment and can be archived.

[0064] The at least one data is transferred from the first storage system to the archival storage system, that is, the at least one data is copied to the archival storage system for storage, and the at least one data in the first storage system is deleted.

[0065] In some embodiments of the present disclosure, when all data in the target storage bucket meets the target condition, the target storage bucket is transferred from the first storage system to the archival storage system.

[0066] In some embodiments of the present disclosure, when part of the data in the target storage bucket meets the target condition, the part of the data in the target storage bucket that meets the target condition is transferred from the first storage system to the archive storage system.

[0067] Among them, a large amount of data is stored in the storage bucket. Archiving management at the granularity of the storage bucket can improve the processing efficiency of converting data into archival storage and reduce storage costs.

[0068] In the embodiment of the present disclosure, the archiving lifecycle is set at the bucket granularity in the first storage system, and then the data in the bucket whose access frequency during the archiving lifecycle is less than or equal to the frequency threshold and / or the access time of the last access during the archiving lifecycle is greater than or equal to the duration threshold from the current time is transferred to the archiving storage system for storage, thereby transferring the data with a low access frequency during the archiving lifecycle and without any access for a long time to the archiving storage, thereby reducing the waste of storage costs to a certain extent. Moreover, setting the archiving lifecycle at the bucket granularity can improve the setting efficiency of the archiving lifecycle compared to setting the archiving lifecycle for each data, thereby improving the archiving processing efficiency of the data and effectively reducing the storage cost.

[0069] In some embodiments of the present disclosure, the above step 101 can be specifically implemented through the following step 101a.

[0070] 101a. When the bucket identifier of the target bucket is not in the bucket blacklist, set an archiving lifecycle for the target bucket in the first storage system.

[0071] The bucket blacklist includes at least one bucket identifier, and the bucket identifier is used to indicate a bucket.

[0072] The bucket identifier may be a bucket number, name, etc., which is not limited here.

[0073] The bucket blacklist may be determined according to the access real-time requirements of the buckets. For example, buckets with high access real-time requirements may be set in the corresponding bucket blacklist.

[0074] In the disclosed embodiment, by setting a bucket blacklist, the corresponding buckets in the bucket blacklist cannot be archived for storage. Therefore, through the bucket blacklist, the buckets in the first storage system that cannot be archived for storage can be quickly excluded, thereby improving the processing efficiency of archiving data at the bucket granularity.

[0075] In some embodiments of the present disclosure, the target condition also includes: the data does not include a data tag in a tag blacklist, the tag blacklist includes at least one data tag, and the data tag is used to indicate that the corresponding data cannot be archived.

[0076] The data label can be determined according to the actual situation and is not limited here. Data with data labels in the blacklist cannot be archived and stored. Therefore, through the label blacklist, data that cannot be archived and stored in the target bucket can be quickly excluded, thereby improving the processing efficiency of archiving and storing data at the bucket granularity.

[0077] Among them, the data (file) granularity is used to archive and check the data that is of high importance and needs to be accessed in real time, such as adding tags (i.e. data tags) in the file metadata to distinguish them from being archived. For example, the metadata of the data in the storage bucket that is of high importance and needs to be accessed in real time and has a high data access frequency is marked with a data tag, indicating that the data including the data tag will never be archived.

[0078] In some embodiments of the present disclosure, it is possible to first determine whether the data in the target storage bucket includes any data tags in the tag blacklist, filter the data in the storage bucket, and then determine whether the access frequency of the filtered data (excluding data with any data tags in the tag blacklist) within the archiving lifecycle is less than or equal to a frequency threshold and / or the access time of the last access within the archiving lifecycle, and determine at least one data.

[0079] In some embodiments of the present disclosure, the target condition includes that the access frequency of the data during the archiving life cycle is less than or equal to a frequency threshold; after the above step 101, the data storage method provided by the embodiment of the present disclosure may also include the following step 103.

[0080] 103. Determine the access frequency of the data within the archiving life cycle according to the data index file or log analysis file corresponding to the target storage bucket.

[0081] In some embodiments of the present disclosure, the access time and cumulative access times of each access to the corresponding data can be recorded in a data index file or a log analysis file, and then the access frequency of the data in the archive life cycle can be determined based on the data index file or the log analysis file. In this way, the access frequency of the data in the archive life cycle can be determined quickly and accurately, and then it can be determined whether the data meets the target condition.

[0082] In some embodiments of the present disclosure, the target condition includes the access time of the last access to the data within the archiving life cycle, and the time length from the current time is greater than or equal to the time length threshold; after the above step 101, the data storage method provided by the embodiment of the present disclosure may also include the following step 104.

[0083] 104. Determine the last access time of the data within the archiving lifecycle according to the data index file or the log analysis file corresponding to the target storage bucket.

[0084] In some embodiments of the present disclosure, the access time and cumulative access times of each access to the corresponding data can be recorded in a data index file or a log analysis file, and then the access time of the last access to the data within the archive life cycle can be determined based on the data index file or the log analysis file. In this way, the access time of the last access to the data within the archive life cycle can be quickly and accurately determined, and then it can be determined whether the data meets the target condition.

[0085] In the disclosed embodiment, a data index file is used to record the access time and cumulative number of accesses of each access to the corresponding data, thereby providing a method for determining the access frequency within the archiving life cycle and / or the access time of the last access within the archiving life cycle based on the data index file, which can improve the processing efficiency of archiving and storing data.

[0086] In some embodiments of the present disclosure, by calling an archiving service interface, the storage buckets in the first storage system and the data in the storage buckets are regularly screened, and the screened data that meets the target conditions is archived.

[0087] In some embodiments of the present disclosure, the archiving lifecycle can be set in the storage bucket in advance, and data tags can be processed for data files, and a storage bucket blacklist and a tag blacklist can be set to ensure that important files and important storage buckets are excluded from the archivable scope; then, by regularly scanning the storage buckets in the first storage system and the data in the storage buckets, the data that meets the target conditions in the storage buckets that are not in the storage bucket blacklist is archived and stored. After the archiving storage is successful, the data to be archived is recorded and the status is updated.

[0088] In order to save costs, after archiving data from cold storage to archive storage, the archive storage files need to be thawed in advance before they can be accessed again. Generally, it takes (half an hour to eight hours, depending on the thawing priority and the size of the data). Only after the thawing is successful can the files be accessed. In order to ensure smooth production processes and not affect production efficiency, the following temporary thawing and permanent thawing solutions are provided.

[0089] In some embodiments of the present disclosure, Figure 1 ,like Figure 2 As shown, after the above step 102, the data storage method provided by the embodiment of the present disclosure may further include the following steps 105 to 108.

[0090] 105. Receive a data access request for target data in the at least one data.

[0091] The data access request carries the duration of maintaining the thaw state corresponding to the target data.

[0092] The higher the thawing priority, the earlier the thawing order, and the shorter the thawing time from receiving the data access request to successful thawing. The thawing time is related to the thawing priority and the size of the data.

[0093] 106. In response to the data access request, copy the target data from the archival storage system to obtain first data.

[0094] 107. Store the first data in a second storage system.

[0095] The second storage system is a storage system in which stored data can be accessed in real time.

[0096] 108. After the thawed state is maintained for a period of time, delete the first data stored in the second storage system.

[0097] The duration for which the thawed state is preserved is determined based on business needs.

[0098] The second storage system is a storage system in which stored data can be accessed in real time.

[0099] Exemplarily, the second storage system includes any one of the following: a hot storage system, a cold storage system, a standard storage system.

[0100] The first storage system and the second storage system may be the same storage system or different storage systems, which is not limited here.

[0101] Exemplarily, the first data obtained by temporary thawing is usually stored in a thermal storage system or a standard storage system, which is not limited here.

[0102] In the embodiment of the present application, after the thawed state is maintained for a certain period of time, the first data is no longer needed for data access requests, so in order to reduce storage costs, the first data can be deleted.

[0103] In some embodiments of the present disclosure, temporary unfreezing of target data can be achieved by calling the unfreezing service interface to execute the above steps 106 to 108. Temporary unfreezing is to copy the data from the archival storage system to the second storage system. The second storage system is a temporary storage system for storing temporarily unfrozen data.

[0104] In the disclosed embodiment, temporary thawing is immediately triggered in response to a data access request, and the target data is thawed by triggering an asynchronous thawing interface.

[0105] In some embodiments of the present disclosure, the data access request also carries a thawing priority for thawing the target data; the above step 106 can be specifically implemented through the following steps 106a and 106b.

[0106] 106a. In response to the data access request, if there are multiple data to be thawed in the archive storage system, determine the thawing order of the target data according to the thawing priority, and the multiple data to be thawed include the target data.

[0107] The thawing priority of the data to be thawed is determined according to the business requirements of the data to be thawed. The more important the data to be thawed is, the higher the thawing priority is. When there are multiple data to be thawed, the higher the thawing priority is, the earlier the thawing order is.

[0108] 106b. Copy the target data from the archival storage system according to the thawing order to obtain first data.

[0109] In some embodiments of the present disclosure, a thawing order that matches the business demand can be determined according to the thawing priority, and the data can be thawed in time so that the business can access the first data in real time.

[0110] In some embodiments of the present disclosure, Figure 2 ,like Figure 3 As shown, after the above step 105, the data storage method provided by the embodiment of the present disclosure may further include the following step 109, and the above step 106 may be specifically implemented by the following step 106c.

[0111] 109. Determine the cumulative number of visits to the target data within a preset time period.

[0112] 106c. When the accumulated access times are less than the times threshold, in response to the data access request, copy the target data from the archival storage system to obtain the first data.

[0113] Among them, the preset duration and number thresholds can be determined according to actual conditions and are not limited here.

[0114] In the embodiment of the present disclosure, if the cumulative number of accesses within the preset time period is less than the number threshold, it means that the access frequency of the target data is not very high, and thus the target data can be unfrozen by temporary unfreezing.

[0115] It is understandable that some data is not important for a period of time, but the archiving strategy may change in another period. At this time, the archived files need to be migrated from the archive to a storage that can be accessed in real time (such as cold storage). At this time, it is necessary to call the permanent thawing interface to thaw the data and move it to the cold storage.

[0116] In some embodiments of the present disclosure, Figure 3 As shown, after the above step 109, the data storage method provided by the embodiment of the present disclosure may further include the following steps 110 to 113.

[0117] 110. When the accumulated access times is greater than or equal to a times threshold, in response to the data access request, copy the target data from the archival storage system to obtain second data.

[0118] 111. Delete the target data stored in the archiving storage system.

[0119] 112. Store the second data in a third storage system.

[0120] The third storage system is a storage system in which stored data can be accessed in real time.

[0121] Exemplarily, the third storage system includes any one of the following: a hot storage system, a cold storage system, a standard storage system.

[0122] The first storage system, the third storage system, and the second storage system may be the same type of storage system or different types of storage systems, which is not limited here.

[0123] Exemplarily, the thawed first data is usually stored in a thermal storage system or a standard storage system, which is not limited here.

[0124] The above steps 110 to 112 are permanent thawing, which means migrating the data from the archive storage system to the third storage system. The third storage system is used to store the permanently thawed data.

[0125] In the embodiment of the present disclosure, if the cumulative number of accesses within the preset time period is greater than or equal to the number threshold, it means that the target data is accessed frequently. Therefore, the target data can be unfrozen by permanent unfreezing to avoid frequent temporary unfreezing.

[0126] In the disclosed embodiment, by calling the thawing interface, the file thawing (temporary thawing or permanent thawing) is triggered. When the file thawing status changes to success, the service can download the file. Before the file archive is thawed, the user needs to be prompted to wait, and a recursive query or retry is added until the data access is successful.

[0127] In some embodiments of the present disclosure, after the above step 105, the data storage method provided by the embodiments of the present disclosure may further include the following step 113.

[0128] 113. When the target data has not been completely unfrozen, a prompt message is returned in response to the data access request, where the prompt message is used to prompt the user to wait for the target data to be unfrozen.

[0129] In the embodiment of the present disclosure, when the target data has not been completely unfrozen, a prompt message is returned in response to the data access request, which can prompt the user to wait for the target data to be unfrozen, thereby improving the user experience.

[0130] In some embodiments of the present disclosure, after the above step 113, the data storage method provided by the embodiments of the present disclosure may further include the following steps 114 and 115.

[0131] 114. The data access request is received again.

[0132] 115. In response to the data access request, when the target data is unfrozen, return the unfrozen target data.

[0133] Among them, when the target data is temporarily unfrozen, the unfrozen target data is the unfrozen first data (that is, the first data stored in the second storage system); when the target data is permanently unfrozen, the unfrozen target data is the unfrozen second data (that is, the second data stored in the third storage system).

[0134] In some embodiments of the present disclosure, the prompt information may also carry the remaining thawing time to prompt the user to send the data access request again after the remaining thawing time has passed.

[0135] In some embodiments of the present disclosure, in order to obtain the thawed target data as soon as possible, data access requests may be sent periodically (ie, recursive query or retry) to determine the current thawing progress, thereby ensuring data access availability.

[0136] In the embodiment of the present disclosure, in response to a data access request received again, when the target data is thawed, the thawed target data is returned, so that the user can download the thawed target data or the production system can perform production processing on the thawed target data.

[0137] Figure 4 FIG. 1 is a structural block diagram of a data storage device according to an embodiment of the present disclosure. Figure 4 As shown, it includes: a setting module 401, which is used to set an archiving life cycle for a target storage bucket in a first storage system, and the first storage system is a storage system in which the stored data can be accessed in real time; a transfer module 402, which is used to transfer the at least one data from the first storage system to the archiving storage system when there is at least one data meeting the target condition in the target storage bucket, and the archiving storage system is a storage system in which the stored data cannot be accessed in real time; wherein the target condition includes at least one of the following: the access frequency of the data in the archiving life cycle is less than or equal to the frequency threshold; the access time of the last access of the data in the archiving life cycle is greater than or equal to the duration threshold from the current time.

[0138] In some embodiments of the present disclosure, the setting module 401 is specifically used to set an archiving lifecycle for a target bucket in a first storage system when the bucket identifier of the target bucket is not in a bucket blacklist; wherein the bucket blacklist includes at least one bucket identifier, which is used to indicate a bucket.

[0139] In some embodiments of the present disclosure, the target condition also includes: the data does not include a data tag in a tag blacklist, the tag blacklist includes at least one data tag, and the data tag is used to indicate that the corresponding data cannot be archived.

[0140] In some embodiments of the present disclosure, the target condition includes that the access frequency of the data during the archiving life cycle is less than or equal to a frequency threshold; the device also includes: a determination module for determining the access frequency of the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket; and / or, the target condition includes the access time of the last access to the data during the archiving life cycle, the duration from the current moment being greater than or equal to a duration threshold; the device also includes: a determination module for determining the access time of the last access to the data during the archiving life cycle based on the data index file or log analysis file corresponding to the target storage bucket.

[0141] In some embodiments of the present disclosure, the device also includes: a receiving module, a copying module, a storage module and a deletion module, the receiving module being used to receive a data access request for target data in the at least one data after transferring the at least one data from the first storage system to the archival storage system, the data access request carrying the thaw state retention time corresponding to the target data; the copying module being used to copy the target data from the archival storage system to obtain the first data in response to the data access request; the storage module being used to store the first data in a second storage system, the second storage system being a storage system in which the stored data can be accessed in real time; the deletion module being used to delete the first data stored in the second storage system after the thaw state is maintained for a certain time.

[0142] In some embodiments of the present disclosure, the data access request also carries a thawing priority for thawing the target data; the copy module is specifically used to respond to the data access request and, when there are multiple data to be thawed in the archive storage system, determine the thawing order of the target data according to the thawing priority, and the multiple data to be thawed include the target data; and copy the target data from the archive storage system according to the thawing order to obtain the first data.

[0143] In some embodiments of the present disclosure, the device also includes: a determination module, used to determine the cumulative number of accesses to the target data within a preset time period after receiving a data access request for the target data in the at least one data; and a copy module, specifically used to copy the target data from the archive storage system to obtain the first data in response to the data access request when the cumulative number of accesses is less than a number threshold.

[0144] In some embodiments of the present disclosure, the copy module is further used to, after determining the cumulative number of accesses to the target data within a preset time period, copy the target data from the archive storage system in response to the data access request to obtain the second data when the cumulative number of accesses is greater than or equal to a threshold number; the device also includes: a deletion module, used to delete the target data stored in the archive storage system; the storage module is also used to store the second data in a third storage system, the third storage system being a storage system in which the stored data can be accessed in real time.

[0145] In some embodiments of the present disclosure, the device also includes: a return module, which is used to receive a data access request for target data in the at least one data, and return a prompt message in response to the data access request if the target data has not been unfrozen, wherein the prompt message is used to prompt the user to wait for the target data to be unfrozen.

[0146] In some embodiments of the present disclosure, the receiving module is further used to receive the data access request again after returning prompt information in response to the data access request; the returning module is further used to return the unfrozen target data in response to the data access request when the target data is unfrozen.

[0147] In the embodiments of the present disclosure, each module can implement the data storage method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0148] Figure 5 A structural diagram of an electronic device provided for an embodiment of the present disclosure is used to exemplarily illustrate an electronic device that implements any data storage method in an embodiment of the present disclosure, and should not be understood as a specific limitation on the embodiment of the present disclosure.

[0149] like Figure 5As shown, the electronic device 500 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0150] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 500 is shown with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0151] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processor 501, the functions defined in any data storage method provided in the embodiment of the present disclosure can be executed.

[0152] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0153] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0154] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0155] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: sets an archiving life cycle for a target storage bucket in a first storage system, where the first storage system is a storage system in which stored data can be accessed in real time; if there is at least one data that meets the target condition in the target storage bucket, transfer the at least one data from the first storage system to an archiving storage system, where the archiving storage system is a storage system in which stored data cannot be accessed in real time; wherein the target condition includes at least one of the following: the access frequency of the data within the archiving life cycle is less than or equal to the frequency threshold; the access time of the last access of the data within the archiving life cycle is greater than or equal to the duration threshold from the current time.

[0156] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0159] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0160] In the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0162] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0163] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data storage method, characterized in that: The method comprises: Setting an archiving lifecycle for a target storage bucket in a first storage system, where the first storage system is a storage system in which stored data can be accessed in real time; If at least one data satisfying the target condition exists in the target storage bucket, the at least one data is transferred from the first storage system to an archival storage system, where the archival storage system is a storage system in which the stored data cannot be accessed in real time; The target condition includes at least one of the following: The access frequency of the data during the archiving life cycle is less than or equal to the frequency threshold; The time from the last access time of the data within the archiving lifecycle to the current time is greater than or equal to the time threshold.

2. The method according to claim 1, characterized in that The step of setting an archiving lifecycle for a target storage bucket in the first storage system includes: If the bucket identifier of the target bucket is not in the bucket blacklist, setting an archiving lifecycle for the target bucket in the first storage system; The bucket blacklist includes at least one bucket identifier, and the bucket identifier is used to indicate a bucket.

3. The method according to claim 1, characterized in that The target conditions also include: The data does not include a data tag in a tag blacklist, and the tag blacklist includes at least one data tag, and the data tag is used to indicate that the corresponding data cannot be archived and stored.

4. The method according to claim 1, characterized in that: The target condition includes that the access frequency of the data in the archiving life cycle is less than or equal to a frequency threshold; the method further includes: Determining the access frequency of the data within the archiving life cycle according to the data index file or log analysis file corresponding to the target storage bucket; and / or, The target condition includes the last access time of the data in the archiving life cycle, and the time length from the current time is greater than or equal to the time length threshold; the method further includes: Determine the access time of the last access to the data within the archiving life cycle according to the data index file or the log analysis file corresponding to the target storage bucket.

5. The method according to claim 1, characterized in that After transferring the at least one data from the first storage system to the archival storage system, the method further comprises: receiving a data access request for target data among the at least one data, wherein the data access request carries a thaw state retention time length corresponding to the target data; In response to the data access request, copying the target data from the archival storage system to obtain first data; storing the first data in a second storage system, where the second storage system is a storage system in which the stored data can be accessed in real time; After the thawed state is maintained for a period of time, the first data stored in the second storage system is deleted.

6. The method according to claim 5, characterized in that The data access request also carries a thawing priority for thawing the target data; In response to the data access request, copying the target data from the archival storage system to obtain the first data includes: In response to the data access request, if there are a plurality of data to be thawed in the archive storage system, determining a thawing order of the target data according to the thawing priority, the plurality of data to be thawed including the target data; The target data is copied from the archival storage system according to the thawing order to obtain the first data.

7. The method according to claim 5, characterized in that After receiving the data access request for the target data in the at least one data, the method further includes: Determine the cumulative number of visits to the target data within a preset time period; In response to the data access request, copying the target data from the archival storage system to obtain the first data includes: When the accumulated access times are less than the times threshold, in response to the data access request, the target data is copied from the archival storage system to obtain the first data.

8. The method according to claim 7, characterized in that After determining the cumulative number of visits to the target data within a preset time period, the method further includes: When the accumulated access times are greater than or equal to a times threshold, in response to the data access request, copying the target data from the archival storage system to obtain second data; Deleting the target data stored in the archiving storage system; The second data is stored in a third storage system, where the third storage system is a storage system in which the stored data can be accessed in real time.

9. The method according to any one of claims 5 to 8, characterized in that After receiving the data access request for the target data in the at least one data, the method further includes: In the case that the target data has not been completely unfrozen, prompt information is returned in response to the data access request, and the prompt information is used to prompt the user to wait for the target data to be unfrozen.

10. The method according to claim 9, characterized in that After returning prompt information in response to the data access request, the method further includes: receiving the data access request again; In response to the data access request, when the target data is completely unfrozen, the unfrozen target data is returned.

11. A data storage device, characterized in that: include: A setting module, used to set an archiving lifecycle for a target storage bucket in a first storage system, where the first storage system is a storage system in which stored data can be accessed in real time; a transfer module, configured to transfer, if at least one data satisfying the target condition exists in the target storage bucket, the at least one data from the first storage system to an archival storage system, where the archival storage system is a storage system in which the stored data cannot be accessed in real time; The target condition includes at least one of the following: The access frequency of the data during the archiving life cycle is less than or equal to the frequency threshold; The time from the last access time of the data within the archiving lifecycle to the current time is greater than or equal to the time threshold.

12. An electronic device, characterized in that: include: A memory and a processor, the memory is used to store a computer program; the processor is used to execute the data storage method described in any one of claims 1 to 10 when calling the computer program.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 10 is implemented.

Images