Data storage method, system and device

Abstract

The invention discloses a data storage method, system and device. Relates to the field of data storage, and comprises the following steps: receiving a plurality of target objects, writing the plurality of target objects into an archiving source barrel, and merging the plurality of target objects into at least one large object through the archiving source barrel; after the storage duration of each large object is larger than or equal to a preset first life cycle, the large objects are moved from an archiving source bucket to an archiving target bucket, and the archiving target bucket is a data bucket with the data processing efficiency lower than that of the archiving source bucket; and scanning the large object in the archiving target bucket every preset duration, and writing the target large object into a backup archiving module after detecting that the storage duration of the target large object is greater than or equal to a preset second life cycle. Through the object storage method and device, the problem that the efficiency is low when a user accesses the stored data due to the fact that the object is directly stored when the object is stored in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data storage, and in particular to a data storage method, system and device. Background Art

[0002] As more and more companies apply large models, their business volume is also growing rapidly. A large number of businesses have led to higher and higher demands for high throughput, large IOPS (Input / Output Operations Per Second), and low latency data access, and they need to ensure fast data processing and transmission. In the process of receiving user data access, the system will face high-speed access to a large number of small files.

[0003] In the related art, data storage does not distinguish between large and small objects, and data is directly archived in the database. When users access small objects stored in the database, they directly read the data from the database. However, this data storage method is difficult to meet performance requirements in scenarios with massive small files and large IOPS. For example, users usually need high-speed read and write performance for objects within 30 days. After 30 days, the data needs to be quickly archived to a warm data pool and retained for 60 days. After that, the data is accessed more frequently. The private cloud or public cloud architecture in the related art directly reads and archives data. Since reading and deleting small objects consumes a lot of resources and is slow, it is difficult to meet the performance requirements and data reading efficiency of scenarios with massive small files and large IOPS.

[0004] Currently, no effective solution has been proposed to the problem that objects are directly stored in the related art, resulting in low efficiency when users access the stored data. Summary of the invention

[0005] The main purpose of the present application is to provide a data storage method, system and device to solve the problem in the related art that objects are directly stored when storing objects, resulting in low efficiency when users access the stored data.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data storage method is provided. The method includes: receiving multiple target objects, writing the multiple target objects into an archive source bucket, merging the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, wherein the archive target bucket is a data bucket with a lower data processing efficiency than the archive source bucket; scanning the large objects in the archive target bucket at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, writing the target large object into a backup archive module.

[0007] Optionally, merging multiple target objects into at least one large object by archiving a source bucket includes: creating an object file in which no data is stored in the archiving source bucket, and moving multiple target objects to the object file one by one; in the process of moving each target object into the object file, determining whether the current storage capacity of the object file is greater than or equal to a second storage capacity threshold; if the current storage capacity of the object file is greater than or equal to the second storage capacity threshold, determining the object file as a large object, creating the next object file, and moving the target objects among the multiple target objects that have not been moved into the large object into the next object file until all target objects are moved into the object file; if the current storage capacity of the object file is less than the second storage capacity threshold, moving the next target object to the object file.

[0008] Optionally, before moving the next target object to the object file, the method further includes: determining whether the number of target objects in the object file is greater than or equal to a preset number; if the number of target objects in the object file is greater than or equal to the preset number, determining the object file as a large object, creating the next object file, and moving target objects among the multiple target objects that have not been moved into the large object into the next object file until all target objects are moved into the object file; if the number of target objects in the object file is less than the preset number, executing the step of moving the next target object to the object file.

[0009] Optionally, after moving the large object from the archiving source bucket to the archiving target bucket, the method further includes: for each large object moved into the archiving target bucket, deleting the large object in the archiving source bucket; and creating backup metadata of the large object in the archiving target bucket based on the metadata of the large object, wherein the backup metadata includes at least one of the following: the name and storage location information of the large object, and the name and storage capacity of the target object.

[0010] Optionally, writing the target large object into the backup archiving module includes: determining whether the target large object is configured with a retrieval method of a backup retrieval mode; if the target large object is not configured with a retrieval method of a backup retrieval mode, writing the target large object into the backup archiving module, and deleting the data of the target large object, the metadata of the target large object, and the backup metadata of the target large object from the archiving target bucket; if the target large object is configured with a retrieval method of a backup retrieval mode, writing the target large object into the backup archiving module, deleting the data of the target large object from the archiving target bucket, and updating the storage location in the metadata of the target large object from the archiving target bucket to the backup archiving module.

[0011] Optionally, the method also includes: when a user's retrieval instruction for a retrieval object is detected, determining a retrieval method of the retrieval instruction, wherein the retrieval object is any one of a plurality of target objects, and the retrieval method includes a backup retrieval mode and an index mode; when the retrieval method is the backup retrieval mode, reading the retrieval object based on the backup metadata; when the retrieval method is the index mode, reading the retrieval object based on the metadata index information.

[0012] Optionally, reading the retrieval object based on the backup metadata includes: when the retrieval object is stored in an archive source bucket, reading the retrieval object from the archive source bucket; when the retrieval object is stored in an archive target bucket, determining the large object to which the retrieval object belongs through the backup metadata in the archive target bucket, and reading the retrieval object from the large object; when the retrieval object is stored in a backup archive module, determining the large object to which the retrieval object belongs through the backup metadata in the archive target bucket, and retrieving and reading the retrieval object from the backup archive module based on the metadata of the large object.

[0013] Optionally, reading the retrieval object based on metadata index information includes: determining the storage location of the retrieval object through the metadata index information; when the storage location is an archive source bucket or an archive target bucket, reading the retrieval object from the archive source bucket or the archive target bucket; when the storage location is a backup archive module, retrieving and reading the retrieval object from the backup archive module through the metadata index information.

[0014] According to another aspect of the present application, a data storage system is provided. The system includes: a data collection client, which is used to receive multiple target objects and write the multiple target objects into an archive source bucket; an archive storage module, which includes an archive source bucket, an archive target bucket and an archive open interface, the archive source bucket is used to merge the multiple target objects into at least one large object, and after the storage time of the large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, the archive open interface scans the large object in the archive target bucket every preset time, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, the target large object is written into a backup archive module; the backup archive module is used to store the target large object.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data storage device is provided. The device includes: a receiving unit, which is used to receive multiple target objects, write the multiple target objects into an archive source bucket, and merge the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; a moving unit, which is used to move the large object from the archive source bucket to the archive target bucket after the storage time of each large object is greater than or equal to a preset first life cycle, wherein the archive target bucket is a data bucket with a lower data processing efficiency than the archive source bucket; and a scanning unit, which is used to scan the large objects in the archive target bucket every preset time, and write the target large object into the backup archive module after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle.

[0016] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the data storage method described in each embodiment of the present application are implemented.

[0017] Through the present application, the following steps are adopted: receiving multiple target objects, writing the multiple target objects into an archive source bucket, merging the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that uses object storage to store data, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, wherein the archive target bucket is a data bucket with lower data processing efficiency than the archive source bucket; scanning the large objects in the archive target bucket at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, writing the target large object into a backup archive module, thereby solving the problem in the related art that objects are directly stored when stored, resulting in low efficiency when users access the stored data. By merging target objects whose storage capacity is less than a first storage capacity threshold into large objects, users can change from random reading to sequential reading when reading target objects. In addition, large objects are flowed among the archive source bucket, the archive target bucket and the backup archive module according to the preset first life cycle and second life cycle, so that data with high user access frequency is stored in the archive source bucket with high data processing efficiency, and data with low access frequency is stored in the archive target bucket with relatively lower data processing efficiency than the archive source bucket. Finally, the data flows into the backup archive module of cold storage, which reduces the performance requirements for data storage and thus achieves the effect of improving the user's reading efficiency of stored data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flowchart of a data storage method provided according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of grouping a target object into a larger object according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of a user searching for a search object according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a data storage system provided according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a data storage device provided according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0029] It should be noted that the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flowchart of a data storage method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S101, receiving multiple target objects, writing the multiple target objects into an archiving source bucket, and merging the multiple target objects into at least one large object through the archiving source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archiving source bucket is a data bucket that stores data using an object storage method, the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold.

[0032] Specifically, the first storage capacity threshold can be set to 4M, and objects with a storage capacity less than 4M are regarded as small objects, that is, target objects. The archive source bucket is a storage bucket used to store large amounts of data for long-term preservation and management. The archive source bucket can be a physical storage device, a cloud storage service, etc. Since storage devices usually have a minimum management unit, for example, the EC (Elastic Compute, elastic computing service) storage scenario will design a stripe of 512KB, 1MB or a higher size (4M or 8M), and the replica will also set the management unit to 512KB. This configuration can improve write efficiency, and higher management strength can reduce the complexity of metadata management. If a 4KB object is written, it will also occupy the minimum management unit space (such as 512KB). By archiving the target object into a large object, space waste can be reduced, or there is no space waste.

[0033] The machine client collects user data in real time to write the target objects produced into the archive source bucket. The archive source bucket merges the target objects into large objects according to the pre-configured preset number and the second storage capacity threshold. For example, the preset number can be set to 10,000, and the second storage capacity threshold can be set to 100M. In the archive source bucket, each time the number of written target objects reaches 10,000 or the sum of the storage capacity of the written target objects exceeds 100M, they are merged into a large object.

[0034] It should be noted that, in addition to the target objects, the objects written into the archive source bucket may also include ordinary objects with a data size exceeding 4M. In addition to merging target objects, ordinary objects may also be merged into large objects. In addition, the target objects that have been merged into large objects may be deleted, resulting in data holes in the large objects. Therefore, for large objects with data holes, data holes can be avoided by merging them into other target objects or ordinary objects.

[0035] Step S102, after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archiving source bucket to the archiving target bucket, wherein the archiving target bucket is a data bucket with lower data processing efficiency than the archiving source bucket.

[0036] Specifically, the first life cycle can be set according to expert experience and can be adjusted at any time. For example, the first life cycle is set to n days. The archiving target bucket is a storage bucket used to store large amounts of data for long-term preservation and management. The archiving target bucket can be a physical storage device, a cloud storage service, etc. Since users frequently access the stored data within a certain period of time, the access frequency becomes medium after a period of time, and finally becomes low. Therefore, it is necessary to support high-performance and fast data flow for the stored data. By configuring the first life cycle of n days for the archiving source bucket, if the storage time of the large object stored in the archiving source bucket exceeds n days, the large object is moved to the archiving target bucket whose data processing efficiency is lower than that of the archiving source bucket.

[0037] It should be noted that large objects are allowed to be retained in the archive source bucket for a certain number of days to enable users to access the target objects efficiently. After the expiration date, the large objects are read and archived to the archive target bucket. The difference between the archive source bucket and the archive target bucket is that they use different resource pools. The archive source bucket is a user business bucket, corresponding to a resource pool suitable for high-speed access, such as a high-speed cache pool composed of SSD (Solid State Drive) or a replica medium-speed cache pool composed of HDD (Hard Disk Drive); the archive target bucket can use a cold resource pool with low-speed access, such as an EC pool composed of HDD, to minimize user usage costs.

[0038] Step S103, scanning the large objects in the target bucket for archiving at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to the preset second life cycle, writing the target large object into the backup archiving module.

[0039] Specifically, the backup and archiving module may include a tape library, and the second life cycle may be set according to expert experience and may be adjusted at any time. For example, the second life cycle is set to m days, and the large objects in the archive target bucket m days ago are periodically scanned at preset time intervals, and the large objects with a storage time greater than or equal to the preset second life cycle are read and written to the tape library through the archive open interface. By merging the target objects into large objects, and realizing the flow between hot data, warm data and cold data according to the first life cycle and the second life cycle through the archive source bucket, the archive target bucket and the backup and archiving module, the data flow speed is increased by merging the target objects into large objects. Through the hierarchical storage of the archive source bucket, the archive target bucket and the backup and archiving module, a process of step-by-step flow of hot data, warm data and cold data is formed, thereby gradually reducing the user's use cost, and finally archiving the data to the cold storage tape library, and the data flow process can flexibly configure the retention time of hot data, warm data and cold data.

[0040] Among them, the archive open interface supports reading, writing, listing and deleting large objects. Hot data, warm data and cold data refer to the concepts of classifying the access frequency and importance of data in data processing and storage. This classification can help organizations manage data effectively to better optimize performance and resource utilization. Hot data refers to data with a high access frequency, which is usually read and written frequently. This type of data is usually in an active state and needs to be accessed and processed quickly. Hot data is usually stored on high-speed storage media, such as memory or fast solid-state drives. Warm data refers to data with a moderate access frequency, which is usually read and written frequently but not as frequently as hot data. Warm data is usually stored on storage media with high performance but low cost, such as solid-state drives or high-speed hard drives. Cold data refers to data with a low access frequency, which is usually infrequently read and written or has expired. This type of data may include historical records, archived data or backup data. Cold data is usually stored on storage media with low performance but low cost, such as tapes or cloud storage.

[0041] The data storage method provided by the embodiment of the present application receives multiple target objects, writes the multiple target objects into an archive source bucket, and merges the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, wherein the archive target bucket is a data bucket with a lower data processing efficiency than the archive source bucket; the large objects in the archive target bucket are scanned at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, the target large object is written into a backup archive module, thereby solving the problem in the related art that objects are directly stored when stored, resulting in low efficiency when users access the stored data. By merging target objects whose storage capacity is less than a first storage capacity threshold into large objects, users can change from random reading to sequential reading when reading target objects. In addition, large objects are flowed among the archive source bucket, the archive target bucket and the backup archive module according to the preset first life cycle and second life cycle, so that data with high user access frequency is stored in the archive source bucket with high data processing efficiency, and data with low access frequency is stored in the archive target bucket with relatively lower data processing efficiency than the archive source bucket. Finally, the data flows into the backup archive module of cold storage, which reduces the performance requirements for data storage and thus achieves the effect of improving the user's reading efficiency of stored data.

[0042] In order to improve the user's access and reading efficiency of the target object, it is necessary to merge the target objects into large objects. Optionally, in the data storage method provided in the embodiment of the present application, merging multiple target objects into at least one large object by archiving the source bucket includes: creating an object file that does not store data in the archiving source bucket, and moving multiple target objects to the object file one by one; in the process of moving each target object into the object file, judging whether the current storage capacity of the object file is greater than or equal to a second storage capacity threshold; when the current storage capacity of the object file is greater than or equal to the second storage capacity threshold, determining the object file as a large object, creating the next object file, and moving the target objects among the multiple target objects that have not been moved into the large object into the next object file until all the target objects are moved into the object file; when the current storage capacity of the object file is less than the second storage capacity threshold, moving the next target object to the object file.

[0043] Specifically, when merging target objects, the merge can be performed according to a preset second storage capacity threshold, by creating an object file that does not store data in the archive source bucket, and moving the target objects written into the archive source bucket to the object file one by one until the current storage capacity of the target objects stored in the object file is greater than or equal to 100M. The object file is no longer moved into the target object, and the object file is generated into a large object. If there are still target objects that have not been merged, create the next object file, and continue to move the remaining target objects to the next object file until all target objects are moved into the object file to generate large objects.

[0044] For example, Figure 2 is a schematic diagram of grouping a target object into a larger object according to an embodiment of the present application, such as Figure 2 As shown, the machine client collects user data in real time to write the target object produced into the archive source bucket, and the archive source bucket merges the target objects of 0 to 4M into large objects. When the target object is written, the access log is recorded, and the archive is performed by scanning the access log records. The access log contains the merged objects (that is, large objects) and ordinary objects larger than 4M. Figure 2 1.pdf, a.txt,...ee.git in the file are multiple small objects (i.e., target objects). These small objects are archived into a merged object. In addition, Figure 2 The film2.mp4 and film3.mp4 in the file can be common objects larger than 4M. If the sum of film2.mp4 and film3.mp4 meets 100M, they can be merged into one large object. If the size of film1.mp4 exceeds 100M, it is directly regarded as one large object.

[0045] After the large object is moved into the archiving target bucket, the archiving large object generates a name for the large object. For example, the large object name of multiple small objects such as 1.pdf, a.txt, ... ee.git is "D / 2024-08-20-8Jhh", the large object name corresponding to film1.mp4 is "D / 2024-08-20-e3sF", and the large object name of film2.mp4 and film3.mp4 is "D / 2024-08-21-e3sF".

[0046] It should be noted that the target object of the merge may be deleted and cause holes in the large object. Therefore, the target object of the merge can generate a large task together with the ordinary object. The judgment method is to continuously scan the large object until 100M or 10,000 target objects are met before archiving the large object.

[0047] Since direct reading of small objects is a random read for the storage device, which is inefficient and wastes cluster resources, this embodiment merges the target objects into large objects so that users can change the reading method from random reading to sequential reading when accessing the stored target objects, thereby improving data archiving efficiency.

[0048] In addition to determining the archived large object by the second storage capacity threshold, the archived large object may also be determined based on the number of target objects. Optionally, in the data storage method provided in the embodiment of the present application, before moving the next target object to the object file, the method further includes: determining whether the number of target objects in the object file is greater than or equal to a preset number; if the number of target objects in the object file is greater than or equal to the preset number, determining the object file as a large object, creating the next object file, and moving target objects among the multiple target objects that have not been moved into the large object into the next object file until all target objects are moved into the object file; if the number of target objects in the object file is less than the preset number, executing the step of moving the next target object to the object file.

[0049] Specifically, archiving a target object as a large object requires meeting certain threshold conditions. The target object is generated as a large object for archiving only when the number of target objects is greater than or equal to the preset number of 10,000 or the size exceeds 100M. Ordinary objects larger than 4M need to be scanned continuously until the threshold is met. Figure 2 As shown, the archived large object contains the data, metadata, number statistics and archived large object name information of m target objects, where the name of the archived large object is used to verify the process of writing a new large object in the future to ensure that the name cannot be modified during the rewriting process of the large object.

[0050] This embodiment merges the target objects into large objects according to their quantity, so that when users access the stored target objects, they can change the reading mode from random reading to sequential reading, thereby improving the archiving efficiency of data.

[0051] In order to facilitate user access to stored data, backup metadata of the large object is created in the archiving target bucket. Optionally, in the data storage method provided in the embodiment of the present application, after the large object is moved from the archiving source bucket to the archiving target bucket, the method further includes: for each large object moved into the archiving target bucket, deleting the large object in the archiving source bucket; and creating backup metadata of the large object in the archiving target bucket based on the metadata of the large object, wherein the backup metadata includes at least one of the following: the name and storage location information of the large object, and the name and storage capacity of the target object.

[0052] Specifically, after the archive source bucket is moved to the archive target bucket with the large object as the force. The archive large object and backup metadata are generated in the archive target bucket. The backup metadata can store the name and size of the target object (that is, the storage capacity of the target object), the name and offset information of the corresponding archive large object (that is, the storage location information). After the large object is archived to the archive target bucket, the large object of the archive source bucket will be deleted. However, when the user accesses the archive source bucket, the archive target bucket can be accessed through the archive source bucket proxy to achieve the purpose of proxy access to the data in the archive target bucket.

[0053] Among them, Figure 2 As shown, the data of the large object in the archiving target bucket includes the archived large object and the archived metadata of the large object. The two are distinguished by prefixes. The archived large object is prefixed with "D", and the archived metadata object is prefixed with "M". The object name is spliced ​​with timestamp information, and users can flexibly choose the time for the data to stay in the archiving target bucket. The format of the archived metadata object is that the archived data object replaces the prefix "D" with "M". The archived large object consists of data and metadata, where the data is the splicing of the data set contained in the large object, and its metadata is all the detailed metadata information merged. The archived metadata only contains metadata, which is equivalent to the archived large object minus the data part.

[0054] For backup metadata, only the most basic metadata information is retained. After the archived large object is written into the archive target bucket, if a large object with the same name is re-archived to the archive target bucket, or the backup metadata of the archived large object is deleted, the statistical information of the archived large object needs to be updated. These statistical information is stored in the meta (i.e., metadata) of the archived large object; at the same time, the archive target bucket must also support the expiration function with large objects as management units, so the metadata information of all large objects needs to be saved in the archive target bucket. When deleting the archived large object in the archive target bucket, if you choose to retain the backup metadata, the archive metadata object of the deleted archived large object will be automatically retained after the archived large object is deleted in the archive target bucket. For example, Table 1 is the structure of the backup metadata stored in the archive target bucket.

[0055] Table 1

[0056]

[0057] This embodiment generates backup metadata for the large object in the archive target bucket, so that the user can use the backup metadata for retrieval when reading the stored target object, thereby improving data reading efficiency.

[0058] When writing the target large object into the backup archiving module, it is determined whether the backup metadata needs to be deleted according to the configured retrieval method. Optionally, in the data storage method provided in the embodiment of the present application, writing the target large object into the backup archiving module includes: judging whether the target large object is configured with the retrieval method of the backup retrieval mode; if the target large object is not configured with the retrieval method of the backup retrieval mode, writing the target large object into the backup archiving module, and deleting the data of the target large object, the metadata of the target large object, and the backup metadata of the target large object from the archiving target bucket; if the target large object is configured with the retrieval method of the backup retrieval mode, writing the target large object into the backup archiving module, deleting the data of the target large object from the archiving target bucket, and updating the storage location in the metadata of the target large object from the archiving target bucket to the backup archiving module.

[0059] Specifically, the backup retrieval mode can be a retrieval method based on backup metadata. The backup archiving module can include a cold storage tape library. After the target large object is written to the tape library, the data of the target large object in the archiving target bucket needs to be deleted. Since the archiving target bucket stores the data, metadata and backup metadata of the large object, before deleting the data of the target large object, it is necessary to first determine whether the target large object is configured with a backup retrieval mode. If the backup retrieval mode is configured, the backup metadata and metadata of the target large object need to be retained and cannot be deleted. Subsequent users need to read the target large object through the backup metadata in the archiving target bucket when reading the target large object through the backup retrieval mode. If the target large object is not configured with a backup retrieval mode, the data of the target large object, the metadata of the target large object and the backup metadata of the target large object in the archiving target bucket are directly deleted.

[0060] This embodiment retains the backup metadata when moving the target large object into the backup archiving module, so that the user can retrieve the target large object based on the backup metadata when reading data, thereby improving data reading efficiency.

[0061] For the stored target object, the user can choose the backup retrieval mode or the index mode to read the data. Optionally, in the data storage method provided in the embodiment of the present application, the method also includes: when a user's retrieval instruction for the retrieval object is detected, determining the retrieval method of the retrieval instruction, wherein the retrieval object is any one of a plurality of target objects, and the retrieval method includes a backup retrieval mode and an index mode; when the retrieval method is the backup retrieval mode, reading the retrieval object based on the backup metadata; when the retrieval method is the index mode, reading the retrieval object based on the metadata index information.

[0062] Specifically, when a user accesses and reads a search object, if the search object exists in an archive source bucket or an archive target bucket, it can be directly read (the archive source bucket can proxy the target object in the archive target bucket), and if the target object is stored in a cold storage backup archive module (such as a tape library), it is necessary to restore the target object from the backup archive module to the archive target bucket, and then support the user to access the search object in the archive target bucket. The search method supports backup search mode or index mode, and the index synchronization module is optional as an alternative to backup metadata, that is, the search method can be configured as index mode, which can meet the needs of storing metadata outside the storage data cluster, so as to achieve the decoupling of object metadata management and data cluster.

[0063] For example, Figure 3 is a schematic diagram of a user searching for a search object according to an embodiment of the present application, such as Figure 3As shown, when the user reads the stored data through the machine client, the user can choose the backup retrieval mode or the index mode. If the backup retrieval mode is selected, the retrieval object is read by searching the backup metadata stored in the backup target bucket. If the index mode is selected, the retrieval object is read by the metadata index information stored in the index synchronization module in the backup archiving module.

[0064] It should be noted that the index mode, as an alternative, reduces the use of metadata by the storage cluster. The index mode is a configuration option for users to choose. If the target object does not generate backup metadata, the index mode is used for retrieval. The index synchronization module is used to store the metadata index information of the object. When the user requests access, the metadata index information is queried first. When the retrieved object exists in the archive source bucket or the archive target bucket, the archive source bucket or the archive target bucket is directly accessed for reading; when the retrieved object is stored in a tape library, the object needs to be pulled back from the tape library for access.

[0065] The design of the index mode takes into account the scenario where most large objects are stored in cold storage, which can reduce one HEAD request (a request method in the HTTP protocol, used to obtain the header information of the server response, rather than requesting the actual content). Because if the backup retrieval mode is used, when the retrieval object is stored in the tape library, the archive source bucket must first send a HEAD request to determine whether the retrieval object is in the archive source bucket or the archive target bucket. If the retrieval object is not in the archive source bucket or the archive target bucket, an additional HEAD object request needs to be sent to the tape library to obtain the retrieval object. For example, if the backup retrieval mode is selected, if the data of the retrieval object is in the cold storage, the first GET (an HTTP request method used to request the server to send data of a specific resource) to read the object returns 404, then a HEAD request needs to be sent to view the large object corresponding to the object, and if it exists, the data retrieval process is triggered. However, if the index mode is selected, only one query is required. However, if the user configures a relatively large number of archiving days n and m, that is, more of the user's data is in the archive source bucket and the archive target bucket, the backup retrieval mode is more suitable, because a GET can directly read the data, while the index mode requires an additional metadata query from the cold storage. It is worth noting that after the backup and archiving module completes the backup and deletes the archived large object in the archiving target bucket, it is necessary to update the metadata information of the large object in the archiving target bucket (including all information in the backup metadata) to the index synchronization module to mark the large object in cold storage.

[0066] This embodiment improves the two retrieval methods, backup retrieval mode and index mode, and is suitable for different data storage scenarios where target objects are stored in large quantities in archive source buckets, archive target buckets, and backup archive modules, and can provide higher data reading efficiency in each storage scenario.

[0067] When reading the retrieval object, a query can be performed through the backup metadata. Optionally, in the data storage method provided in the embodiment of the present application, reading the retrieval object based on the backup metadata includes: when the retrieval object is stored in the archive source bucket, reading the retrieval object from the archive source bucket; when the retrieval object is stored in the archive target bucket, determining the large object to which the retrieval object belongs through the backup metadata in the archive target bucket, and reading the retrieval object from the large object; when the retrieval object is stored in the backup archive module, determining the large object to which the retrieval object belongs through the backup metadata in the archive target bucket, and retrieving and reading the retrieval object from the backup archive module based on the metadata of the large object.

[0068] Specifically, the retrieval object can be a target object. When the retrieval method is a backup retrieval mode, if the retrieval object is stored in an archive source bucket, the retrieval object is directly read through the archive source bucket. If the retrieval object is stored in an archive target bucket, the large object to which the retrieval object belongs is determined based on the backup metadata, and the retrieval object is read from the large object. If the retrieval object has been moved into a backup archive module for cold storage, it is necessary to determine the large object to which the retrieval object belongs, and then retrieve and read the retrieval object from the backup archive module based on the metadata of the large object.

[0069] For example, taking the 1.txt target object as the retrieval object, when the retrieval method is the backup retrieval mode, first perform GET to read the data in the archive source bucket. If the data does not exist in the archive source bucket, perform HEAD to view the big object name big_obj_name corresponding to the backup metadata of the retrieval object. The metadata structure in the backup metadata refers to Table 1. If the big object to which the retrieval object belongs is stored in the backup archive module, a task is sent to the backup archive module to retrieve the data; if the retrieval object exists in the archive target bucket, the retrieval object is read and returned according to the data offset (data_off) and data length (data_size) in the metadata.

[0070] Table 2 may be a list of interfaces opened by the archive target bucket.

[0071] Table 2

[0072]

[0073] The user can query the name of the large object by calling the "Small Object Corresponding Large Object Query" interface. After the query, the corresponding large object is read from the tape library and written into the archive target bucket, and then the user is notified to read the data.

[0074] This embodiment searches for the target object through the backup metadata, and can efficiently read data from the archive source bucket and the archive target bucket, thereby improving the efficiency of the user's search for the target object.

[0075] When reading the retrieval object, you can also query through the metadata index information. Optionally, in the data storage method provided in the embodiment of the present application, reading the retrieval object based on the metadata index information includes: determining the storage location of the retrieval object through the metadata index information; when the storage location is an archive source bucket or an archive target bucket, reading the retrieval object from the archive source bucket or the archive target bucket; when the storage location is a backup archive module, retrieving and reading the retrieval object from the backup archive module through the metadata index information.

[0076] Specifically, if the search method is index mode, such as Figure 3 As shown, the user's retrieval request will first read the index synchronization module from the backup and archive module to obtain metadata information, and determine the storage location of the retrieval object through the metadata index information. If the retrieval object is stored in the archive source bucket, the retrieval object is directly read through the archive source bucket. If the retrieval object exists in cold storage, a task is sent to the backup and archive module to retrieve the data; if the retrieval object exists in the archive target bucket, it is read according to the data offset and length of the retrieval object in the metadata index information.

[0077] This embodiment searches for a target object through metadata index information, and can efficiently determine the storage location of the search object, thereby improving the efficiency of a user's search for the target object.

[0078] According to another aspect of the present application, a data storage system is provided. Figure 4 Schematic diagram of a data storage system provided according to an embodiment of the present application. Figure 4 As shown, the system includes:

[0079] The data collection client is used to receive multiple target objects and write the multiple target objects into the archive source bucket.

[0080] Specifically, the data collection client may be a machine client, and the data storage system collects data in real time to obtain the target objects produced by the machine client. All the collected target objects are written into the archive source bucket.

[0081] The archive storage module includes an archive source bucket, an archive target bucket, and an archive open interface. The archive source bucket is used to merge multiple target objects into at least one large object. After the storage time of the large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket. The archive open interface scans the large objects in the archive target bucket every preset time, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, the target large object is written into the backup archive module.

[0082] Specifically, the archive storage module consists of an archive source bucket, an archive target bucket, and an archive open interface; the archive source bucket is used to store the hot data of the target object within the first life cycle, and supports high-frequency access. In order to change the random reading mode of the target object to a sequential reading mode to improve data reading efficiency, the target object is merged into a large object in the archive source bucket. When the storage time of the large object is greater than or equal to the preset first life cycle, the large object in the archive source bucket is written to the archive target bucket. The archive target bucket is used to store warm data in the form of large objects during the second life cycle, and supports access that is relatively lower than the access frequency of the archive source bucket. When the storage time of the large object is greater than or equal to the preset second life cycle, the large object in the archive target bucket is written to the backup archive module through the archive open interface.

[0083] By designing and defining the archive open interface, access operations to archive large objects can be realized. The interface includes the enumeration, reading, deletion and writing operations of archive large objects. The process is divided into the backup process and the recovery process. For the archive process, the backup client calls the large object enumeration interface, scans and enumerates the archive large objects in the archive target bucket, then reads the archive large object data and writes it to the tape library, then deletes the archive large object in the archive target bucket, and chooses whether to retain the backup metadata according to the configuration mode.

[0084] The backup archive module is used to store target large objects.

[0085] Specifically, the backup and archiving module consists of an index synchronization module, a backup module, and cold storage. The backup and archiving module periodically scans the large objects in the archive target bucket, reads the large objects through the "archive open interface", and writes them to the tape library (i.e., cold storage). During the process of user access to read and retrieve objects, if the data exists in the archive source bucket or the archive target bucket, it can be read directly. If there is a backup and archiving module, it needs to be restored and accessed from the corresponding cold storage (such as a tape library). The index synchronization module in the backup and archiving module is used to support users in retrieving target objects by index.

[0086] This embodiment is applied to distributed storage services through a data storage system. In solving the scenario of backing up and archiving a large number of small files, it can improve archiving efficiency and reduce usage costs. Archiving archived large objects can change random reads into sequential reads, improving archiving efficiency. Through hierarchical storage, the flow between hot data, warm data, and cold data is formed, gradually reducing the user's usage cost, and finally archiving the data to cold storage. The data flow process can flexibly configure the storage time of hot data, warm data, and cold data.

[0087] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0088] The embodiment of the present application also provides a data storage device. It should be noted that the data storage device of the embodiment of the present application can be used to execute the data storage method provided by the embodiment of the present application. The data storage device provided by the embodiment of the present application is introduced below.

[0089] Figure 5 Schematic diagram of a data storage device provided according to an embodiment of the present application. Figure 5 As shown, the device comprises:

[0090] A receiving unit 501 is used to receive multiple target objects, write the multiple target objects into an archiving source bucket, and merge the multiple target objects into at least one large object through the archiving source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archiving source bucket is a data bucket that stores data using an object storage method, and the large object is an object obtained by merging a preset number of target objects, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold;

[0091] A moving unit 502 is used to move the large object from the archiving source bucket to the archiving target bucket after the storage time of each large object is greater than or equal to a preset first life cycle, wherein the archiving target bucket is a data bucket with a lower data processing efficiency than the archiving source bucket;

[0092] The scanning unit 503 is used to scan the large object in the archive target bucket at preset time intervals, and write the target large object into the backup archive module after detecting that the storage time of the target large object is greater than or equal to the preset second life cycle.

[0093] The data storage device provided in the embodiment of the present application receives multiple target objects through a receiving unit 501, writes the multiple target objects into an archive source bucket, and merges the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; the moving unit 502 moves the large object from the archive source bucket to the archive target bucket after the storage time of each large object is greater than or equal to a preset first life cycle, wherein the archive target bucket is a data bucket whose data processing efficiency is lower than that of the archive source bucket; the scanning unit 503 scans the large objects in the archive target bucket at preset time intervals, and upon detecting that the storage time of the target large object is greater than or equal to a preset first life cycle, After a life span equal to or greater than the preset second life cycle, the target large object is written into the backup archiving module, which solves the problem in the related art that objects are directly stored when stored, resulting in low efficiency when users access the stored data. By merging target objects with storage capacity less than the first storage capacity threshold into large objects, users can change from random reading to sequential reading when reading the target objects. In addition, the large object flows between the archiving source bucket, the archiving target bucket and the backup archiving module according to the preset first life cycle and second life cycle, so that data with high user access frequency is stored in the archiving source bucket with high data processing efficiency, and data with low access frequency is stored in the archiving target bucket with relatively lower data processing efficiency than the archiving source bucket. Finally, the data flows into the backup archiving module of cold storage, reducing the performance requirements for data storage, thereby achieving the effect of improving the user's reading efficiency of stored data.

[0094] Optionally, in the data storage device provided in the embodiment of the present application, the receiving unit 501 includes: a first moving module, used to create an object file in which no data is stored in the archive source bucket, and move multiple target objects to the object file one by one; a first judging module, used to judge whether the current storage capacity of the object file is greater than or equal to a second storage capacity threshold in the process of moving each target object into the object file; a first determining module, used to determine the object file as a large object and create the next object file when the current storage capacity of the object file is greater than or equal to the second storage capacity threshold, and move the target objects among the multiple target objects that have not been moved into the large object into the next object file until all the target objects are moved into the object file; a second moving module, used to move the next target object to the object file when the current storage capacity of the object file is less than the second storage capacity threshold.

[0095] Optionally, in the data storage device provided in the embodiment of the present application, the receiving unit 501 also includes: a second judgment module, used to judge whether the number of target objects in the object file is greater than or equal to a preset number; a second determination module, used to determine the object file as a large object when the number of target objects in the object file is greater than or equal to the preset number, and create the next object file, and move the target objects among the multiple target objects that have not been moved into the large object into the next object file until all the target objects are moved into the object file; an execution module, used to execute the step of moving the next target object to the object file when the number of target objects in the object file is less than the preset number.

[0096] Optionally, in the data storage device provided in the embodiment of the present application, the device also includes: a deletion unit, used to delete the large object in the archiving source bucket for each large object moved into the archiving target bucket; a creation unit, used to create backup metadata of the large object in the archiving target bucket based on the metadata of the large object, wherein the backup metadata includes at least one of the following: the name and storage location information of the large object, the name and storage capacity of the target object.

[0097] Optionally, in the data storage device provided in the embodiment of the present application, the scanning unit 503 includes: a third judgment module, used to judge whether the target large object is configured with the retrieval method of the backup retrieval mode; a first writing module, used to write the target large object into the backup archiving module when the target large object is not configured with the retrieval method of the backup retrieval mode, and delete the data of the target large object, the metadata of the target large object and the backup metadata of the target large object from the archiving target bucket; a second writing module, used to write the target large object into the backup archiving module when the target large object is configured with the retrieval method of the backup retrieval mode, delete the data of the target large object from the archiving target bucket, and update the storage location in the metadata of the target large object from the archiving target bucket to the backup archiving module.

[0098] Optionally, in the data storage device provided in the embodiment of the present application, the device also includes: a determination unit, used to determine the retrieval method of the retrieval instruction when a user's retrieval instruction for the retrieval object is detected, wherein the retrieval object is any one of a plurality of target objects, and the retrieval method includes a backup retrieval mode and an index mode; a first reading unit, used to read the retrieval object based on the backup metadata when the retrieval method is the backup retrieval mode; and a second reading unit, used to read the retrieval object based on the metadata index information when the retrieval method is the index mode.

[0099] Optionally, in the data storage device provided in the embodiment of the present application, the first reading unit includes: a first reading module, used to read the retrieval object from the archive source bucket when the retrieval object is stored in the archive source bucket; a third determination module, used to determine the large object to which the retrieval object belongs through the backup metadata in the archive target bucket when the retrieval object is stored in the archive target bucket, and read the retrieval object from the large object; a fourth determination module, used to determine the large object to which the retrieval object belongs through the backup metadata in the archive target bucket when the retrieval object is stored in the backup archive module, and retrieve and read the retrieval object from the backup archive module based on the metadata of the large object.

[0100] Optionally, in the data storage device provided in the embodiment of the present application, the second reading unit includes: a fifth determination module, used to determine the storage location of the retrieval object through metadata index information; a second reading module, used to read the retrieval object from the archive source bucket or the archive target bucket when the storage location is an archive source bucket or an archive target bucket; and a third reading module, used to retrieve and read the retrieval object from the backup archive module through metadata index information when the storage location is a backup archive module.

[0101] The data storage device includes a processor and a memory. The receiving unit 501, the moving unit 502 and the scanning unit 503 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0102] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the efficiency of reading stored data can be improved by adjusting kernel parameters.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0104] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data storage method is implemented.

[0105] An embodiment of the present invention provides a processor, which is used to run a program, wherein a data storage method is executed when the program is running.

[0106] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. Figure 6As shown, the electronic device 601 includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: receiving multiple target objects, writing the multiple target objects into an archive source bucket, merging the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket storing data using an object storage method, the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, wherein the archive target bucket is a data bucket with a lower data processing efficiency than the archive source bucket; scanning the large objects in the archive target bucket at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, writing the target large object into the backup archive module. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0107] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: receiving multiple target objects, writing the multiple target objects into an archive source bucket, merging the multiple target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, and the large object is an object after a preset number of target objects are merged, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; after the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, wherein the archive target bucket is a data bucket whose data processing efficiency is lower than that of the archive source bucket; scanning the large objects in the archive target bucket at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, writing the target large object into a backup archive module.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0117] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data storage method, characterized in that: include: Receiving multiple target objects, writing the multiple target objects into an archiving source bucket, and merging the multiple target objects into at least one large object through the archiving source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archiving source bucket is a data bucket that stores data using an object storage method, the large object is an object obtained by merging a preset number of target objects, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; After the storage time of each large object is greater than or equal to a preset first life cycle, the large object is moved from the archiving source bucket to the archiving target bucket, wherein the archiving target bucket is a data bucket with a lower data processing efficiency than the archiving source bucket; The large object in the archiving target bucket is scanned at preset time intervals, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, the target large object is written into the backup archiving module.

2. The method according to claim 1, characterized in that Merging the plurality of target objects into at least one large object through the archiving source bucket includes: Creating an object file in which no data is stored in the archive source bucket, and moving the multiple target objects one by one to the object file; During the process of moving each target object into the object file, determining whether the current storage capacity of the object file is greater than or equal to the second storage capacity threshold; In a case where the current storage capacity of the object file is greater than or equal to the second storage capacity threshold, determining the object file as a large object, creating a next object file, and moving target objects among the multiple target objects that have not been moved into the large object into the next object file until all target objects are moved into the object file; When the current storage capacity of the object file is less than the second storage capacity threshold, the next target object is moved to the object file.

3. The method according to claim 2, characterized in that Before moving the next target object to the object file, the method further includes: Determine whether the number of target objects in the object file is greater than or equal to the preset number; When the number of target objects in the object file is greater than or equal to the preset number, the object file is determined as a large object, a next object file is created, and target objects among the multiple target objects that have not been moved into the large object are moved into the next object file until all target objects are moved into the object file; In a case where the number of target objects in the object file is less than the preset number, a step of moving the next target object to the object file is performed.

4. The method according to claim 1, characterized in that: After moving the large object from the archive source bucket to the archive target bucket, the method further includes: For each large object moved into the archiving target bucket, delete the large object in the archiving source bucket; The backup metadata of the large object is created in the archiving target bucket based on the metadata of the large object, wherein the backup metadata includes at least one of the following: the name and storage location information of the large object, and the name and storage capacity of the target object.

5. The method according to claim 1, characterized in that Writing the target large object into the backup archive module includes: Determine whether the target large object is configured with a retrieval method of a backup retrieval mode; In the case that the target large object is not configured with the retrieval mode of the backup retrieval mode, the target large object is written into a backup archiving module, and the data of the target large object, the metadata of the target large object and the backup metadata of the target large object are deleted from the archiving target bucket; In the case where the target large object is configured with the retrieval method of the backup retrieval mode, the target large object is written into the backup archiving module, the data of the target large object is deleted from the archiving target bucket, and the storage location in the metadata of the target large object is updated from the archiving target bucket to the backup archiving module.

6. The method according to claim 1, characterized in that The method further comprises: In the case of detecting a user's retrieval instruction for a retrieval object, determining a retrieval mode of the retrieval instruction, wherein the retrieval object is any one of the multiple target objects, and the retrieval mode includes a backup retrieval mode and an index mode; When the retrieval mode is a backup retrieval mode, reading the retrieval object based on the backup metadata; When the search method is the index mode, the search object is read based on metadata index information.

7. The method according to claim 6, characterized in that Reading the retrieval object based on the backup metadata includes: In a case where the search object is stored in the archive source bucket, reading the search object from the archive source bucket; In the case where the search object is stored in the archiving target bucket, determining the large object to which the search object belongs through the backup metadata in the archiving target bucket, and reading the search object from the large object; In the case where the retrieval object is stored in the backup archiving module, the large object to which the retrieval object belongs is determined through the backup metadata in the archiving target bucket, and the retrieval object is retrieved and read from the backup archiving module based on the metadata of the large object.

8. The method according to claim 6, characterized in that Reading the search object based on the metadata index information includes: Determine the storage location of the search object through the metadata index information; In a case where the storage location is the archiving source bucket or the archiving target bucket, reading the retrieval object from the archiving source bucket or the archiving target bucket; In a case where the storage location is the backup archive module, the retrieval object is retrieved and read from the backup archive module through the metadata index information.

9. A data storage system, characterized in that: include: A data collection client, used for receiving a plurality of target objects and writing the plurality of target objects into an archive source bucket; The archive storage module comprises the archive source bucket, the archive target bucket and the archive open interface, the archive source bucket is used to merge the multiple target objects into at least one large object, and after the storage time of the large object is greater than or equal to a preset first life cycle, the large object is moved from the archive source bucket to the archive target bucket, and the archive open interface scans the large objects in the archive target bucket at a preset time interval, and after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle, the target large object is written into the backup archive module; The backup and archiving module is used to store the target large object.

10. A data storage device, characterized in that: include: A receiving unit, configured to receive a plurality of target objects, write the plurality of target objects into an archive source bucket, and merge the plurality of target objects into at least one large object through the archive source bucket, wherein the target object is data whose data storage capacity is less than a first storage capacity threshold, the archive source bucket is a data bucket that stores data using an object storage method, the large object is an object obtained by merging a preset number of target objects, or the large object is an object whose data storage capacity is greater than or equal to a second storage capacity threshold; A moving unit, configured to move each large object from the archiving source bucket to an archiving target bucket after the storage time of each large object is greater than or equal to a preset first life cycle, wherein the archiving target bucket is a data bucket having a lower data processing efficiency than the archiving source bucket; The scanning unit is used to scan the large object in the archiving target bucket at preset time intervals, and write the target large object into the backup archiving module after detecting that the storage time of the target large object is greater than or equal to a preset second life cycle.

Images