Data storage method, device, equipment, medium and program product

Abstract

This application discloses a data storage method, apparatus, device, medium, and program product, relating to the field of computer storage technology. The data storage method of this application includes: receiving a bucket creation instruction and a bucket allocation strategy; creating a main bucket and at least one directory bucket according to the bucket allocation strategy, wherein the bucket allocation strategy is used to specify the objects corresponding to each directory bucket, the directory buckets are used to store the metadata of the corresponding objects, and the main bucket is used to store the entity data of each object; in response to receiving a storage instruction for a first object, searching for the first directory bucket corresponding to the first object, saving the metadata of the first object to the first directory bucket, and saving the entity data of the first object to the main bucket. This application, when dealing with large-scale data storage, can significantly improve the single-bucket storage capacity and data processing efficiency based on the bucket allocation strategy.

Description

Technical Field

[0001] This application relates to the field of computer storage technology, and in particular to data storage methods, apparatus, devices, media and program products. Background Technology

[0002] In traditional object storage systems, buckets are widely used as logical storage containers. Index shards are the basic logical units of a bucket. The core principle of bucket storage technology is to store the metadata and entity data of stored objects together in index shards, and then manage the data uniformly through buckets. Bucket storage technology has become a major solution for distributed data storage.

[0003] Currently, in some technologies, the bucket structure limits the data storage capacity of a single bucket in large-scale data scenarios. While adding virtual buckets can increase bucket storage capacity, it significantly increases data enumeration time. Furthermore, the strong dependency between virtual buckets and bucket directory structures leads to deep coupling between metadata and entity data. Therefore, there is an urgent need for a single-bucket storage method that can increase single-bucket storage capacity and improve data processing efficiency. Summary of the Invention

[0004] This application provides a data storage method, a data storage device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problems of limited single-bucket capacity and low data processing efficiency in related technologies.

[0005] This application provides a data storage method, including:

[0006] Receive bucket creation instructions and bucket allocation policies;

[0007] Based on the bucket allocation strategy, a primary bucket and at least one directory bucket are created. The bucket allocation strategy is used to specify the objects corresponding to each directory bucket, the target bucket is used to store the metadata of the corresponding objects, and the primary bucket is used to store the entity data of each object.

[0008] In response to receiving a storage instruction for the first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket.

[0009] This application also provides a data storage device, including:

[0010] The instruction receiving module is used to receive bucket creation instructions and bucket allocation policies;

[0011] The bucket creation module is used to create a main bucket and at least one directory bucket according to the bucket allocation strategy. The bucket allocation strategy is used to specify the objects corresponding to each directory bucket, the target bucket is used to store the metadata of the corresponding objects, and the main bucket is used to store the entity data of each object.

[0012] The data storage module is used to respond to a storage instruction received for the first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the data storage method described above when executing the computer program.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data storage methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data storage methods.

[0016] In some embodiments of this application, a data storage bucket allocation strategy is pre-configured, creating a primary bucket and at least one directory bucket. The primary bucket stores the entity data of all objects, and the directory bucket stores the metadata of the corresponding objects. Then, by receiving a storage instruction for a stored object, the system searches for the directory bucket corresponding to the stored object. Based on the found directory bucket, the system can store the entity data of the stored object in the primary bucket and save the metadata of the stored object to the corresponding directory bucket. This application achieves a data storage method for hundreds of billions of data points in a single bucket by decoupling the management of metadata and entity data. This can increase the storage capacity of a single bucket, thereby enabling scientific data planning and storage when facing different business types and improving data processing efficiency. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the data storage method provided in an embodiment of this application;

[0019] Figure 2A flowchart illustrating another data storage method provided in an embodiment of this application;

[0020] Figure 3 A flowchart illustrating yet another data storage method provided in an embodiment of this application;

[0021] Figure 4 Schematic diagram of a data storage device provided for some embodiments of this application;

[0022] Figure 5 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0024] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0025] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Currently, some bucket storage technologies create virtual buckets and expand the number of shards under each virtual bucket. This allows for unified data management while meeting storage needs. However, these technologies have the following problems:

[0027] 1) Single-bucket capacity limitations pose development and maintenance challenges for storing hundreds of billions of data points. Due to the inherent structure of the bucket, the maximum number of objects a single bucket can store is limited to 100 million. Specifically, a single bucket's index shard can only record a maximum of 100,000 object records. When dealing with data storage scales of hundreds of billions, multiple buckets need to be created for managing, accessing, and maintaining the stored objects. This significantly increases the difficulty and cost of development and maintenance.

[0028] 2) Over-reliance on bucket directory structure. When the amount of data in a bucket exceeds a certain threshold, it is classified and layered according to the directory structure of the stored objects, and the storage capacity can be expanded by creating new virtual buckets. However, this may lead to a situation where there are too many directories in a certain bucket and very few stored objects. In this case, the stored objects in the bucket depend on the directory structure, which makes it difficult to store and retrieve subsequent objects.

[0029] 3) High data coupling within buckets makes managing hundreds of billions of data points difficult. This architecture mixes object metadata (such as object name, size, and physical address) and entity data (such as file content) in the same bucket, managing all data uniformly through a single index (such as a hash table or B+ tree). However, as data volume increases, it becomes difficult to flexibly allocate load or optimize the storage structure, leading to increased complexity in managing stored objects.

[0030] In view of this, this application provides a data storage method that can solve the above problems. The data storage method can be applied to a data storage system, or to an electronic device within a data storage system. The electronic device may include, but is not limited to, desktop computers, tablets, laptops, servers, etc.

[0031] See also Figure 1 This is a flowchart illustrating a data storage method provided in some embodiments of this application. Figure 1 In this context, the data storage method includes the following steps:

[0032] Step S101: Receive bucket creation instructions and bucket allocation policies.

[0033] Specifically, after a user clicks the "Create New Bucket" button on the bucket creation page, the electronic device using the method described in this application can send a bucket creation command. The electronic device using the method described in this application can determine whether to receive the bucket creation command by monitoring the status of the "Create New Bucket" button.

[0034] Based on different business plans, users can configure bucket allocation strategies according to the page returned by the electronic device in this application after receiving the bucket creation instruction.

[0035] Step S102: Based on the bucket allocation strategy, create a primary bucket and at least one directory bucket. The bucket allocation strategy is used to specify the objects corresponding to each directory bucket. The directory bucket is used to store the metadata of the corresponding objects, and the primary bucket is used to store the entity data of each object.

[0036] Specifically, after the user configures the bucket allocation strategy according to their own business plan, the electronic device of this application method creates a primary bucket and at least one directory bucket according to the configured bucket allocation strategy. The storage objects corresponding to the created directory buckets are specified according to the bucket allocation strategy. The primary bucket is used to store the entity data of each storage object, and the directory bucket is used to store the metadata of the corresponding storage object.

[0037] In this embodiment, the stored objects may include, but are not limited to, text files, image files, video files, etc. After the bucket allocation strategy specifies the above-mentioned stored objects corresponding to the directory buckets, the main bucket is used to store the entity data of the above-mentioned stored objects (such as the binary data of the above-mentioned stored objects), and the directory buckets are used to store the metadata of the above-mentioned stored objects (such as the name, size, creation time, modification time, etc. of the stored objects).

[0038] Step S103: In response to receiving a storage instruction for the first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket.

[0039] Specifically, when a bucket allocation strategy specifies the storage objects corresponding to each directory bucket, the strategy can define different rules. For example, allocation can be based on the prefix of the storage object name (e.g., metadata for objects whose names begin with "A" is stored in directory bucket A, and metadata for objects whose names begin with "B" is stored in directory bucket B); allocation can be based on hash values ​​(e.g., hash calculation is performed based on the storage object identifier, and the hash result is mapped to different directory buckets); or allocation can be based on business type (e.g., metadata for different applications is stored in different directory buckets). Upon receiving a storage instruction for the first storage object, the corresponding first directory bucket is located according to the rules defined by the bucket allocation strategy, and the metadata of the first storage object is saved to the first directory bucket, while the entity data of the first storage object is saved to the main bucket.

[0040] In some embodiments of this application, the technical solutions involve receiving a user-triggered bucket creation command and a bucket allocation strategy configured based on business needs. Then, a main bucket for storing entities and at least one directory bucket for storing metadata are created according to the bucket allocation strategy. Upon receiving a storage request from a storage object, the corresponding directory bucket is located according to a preset bucket allocation strategy, and the metadata and entity data are written independently. Thus, based on the bucket allocation strategy, a data storage method that separates metadata from entity data can be obtained. This application, through a user-defined bucket allocation strategy and a method of decoupling the metadata and entity data of storage objects, can achieve storage requirements of hundreds of billions per bucket, thereby realizing refined management of storage resources and efficient management of stored data.

[0041] In some embodiments, in conjunction with reference Figure 2 This is a flowchart illustrating another data storage method provided in an embodiment of this application. Figure 2 The data storage method also includes the following steps:

[0042] Step S201: Receive bucket creation instructions and bucket allocation policy. See details below. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0043] Step S202: Based on the bucket allocation strategy, create a primary bucket and at least one directory bucket. The bucket allocation strategy specifies the objects corresponding to each directory bucket. The directory buckets store the metadata of their respective objects, and the primary bucket stores the entity data of each object. (See also...) Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0044] Step S203: In response to receiving a storage instruction for the first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket. See also... Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0045] Step S204: Receive a data enumeration request, which is used to indicate the second object to be displayed.

[0046] Specifically, the process involves receiving a data enumeration request from the client, which contains range information for the second storage object to be displayed. This range information includes, but is not limited to, bucket name, object name prefix, and starting object identifier.

[0047] In this embodiment, the user initiates a data enumeration request through the electronic device described in this application. For example, after the user clicks the "View File List" button on a web page, the front-end page constructs and sends a file list enumeration request to the back-end storage system.

[0048] Step S205: Locate the second directory bucket corresponding to the second object, and retrieve the metadata of the second object from the second directory bucket.

[0049] Specifically, based on the object scope information of the data enumeration request and in conjunction with the pre-configured bucket allocation strategy, one or more second directory buckets corresponding to the second object to be displayed are determined. For example, if the bucket allocation strategy is set to "allocate according to object name prefix", the electronic device of this application method can extract the Prefix field (which is used to add a specified string, such as "doc / ") from the data enumeration request, and then locate the directory bucket involving the object name prefix (such as "dir-bucket-doc").

[0050] In this embodiment, after finding the second directory bucket corresponding to the second object, the electronic device of this application method reads the metadata of the second object through the second directory bucket.

[0051] Step S206: Display the second object based on its metadata.

[0052] Specifically, based on the metadata of the found second object, the obtained metadata is formatted and can be returned to the client by generating a structured response message (such as JSON or XML format).

[0053] In this embodiment, different methods can be selected to display the data. For example, in a web interface, fields such as object name, size, and modification time can be displayed in list form. The display order of the data can also be changed. For example, it supports sorting and filtering based on metadata (such as sorting by name in ascending or descending order, or by modification time in ascending or descending order).

[0054] In the above embodiments, by receiving data enumeration requests and combining them with bucket allocation strategies, the directory bucket corresponding to the target object can be quickly located, and the metadata of the target object can be obtained from that directory bucket, ultimately realizing the display of the target object information. This allows for precise retrieval of the target object based on user intent, improving response efficiency, significantly reducing I / O load and response latency, and simultaneously achieving rapid enumeration of target object information, making it particularly suitable for large-scale object enumeration scenarios.

[0055] In some embodiments, in conjunction with reference Figure 3 This is a flowchart illustrating another data storage method provided in an embodiment of this application. Figure 3 In step S206 above, displaying the second object based on the metadata of the second object also includes, but is not limited to, steps S301 to S303:

[0056] Step S301: Obtain the attribute data of the second object from the metadata of the second object and display the attribute data.

[0057] Specifically, after obtaining the metadata of the second object in step S205, the electronic device of this application obtains the attribute data of the second object based on the metadata. The attribute data includes, but is not limited to, attribute information such as the entity data storage address, storage category, and access permissions corresponding to the metadata.

[0058] In this embodiment, the aforementioned attribute information can be structured and displayed through a front-end interface or API response. For example, on a mobile device, key attribute information can be displayed in the form of cards.

[0059] In step S302, in response to the selection of the attribute data of the second object, the entity data storage address of the second object is obtained from the metadata of the second object.

[0060] Specifically, when a user selects the attribute data of a second object through some means (such as clicking, double-clicking, or long-pressing) (e.g., clicking a file name in a list), the electronic device of this application extracts the entity data storage address of the second object through the metadata of the second object. The entity data storage address may include, but is not limited to, the identification information of the main bucket, the key-value pair of the object in the main bucket, etc.

[0061] Step S303: Obtain and display the entity data of the second object based on the entity data storage address.

[0062] Specifically, by obtaining the entity data storage address of the second object, a data read request is sent to the main bucket to obtain the entity data of the second object (such as binary data like text content, images, and video streams). Then, different processing is performed according to different types of entity data. For example, for text content, online preview or download to local storage is supported; for image content, thumbnails can be generated or the original image can be loaded; for video streams, online viewing is possible through streaming or a built-in player.

[0063] In the above embodiments, by dividing the object display process into three stages—attribute display, address resolution, and entity loading—the architecture of separating the main bucket and directory buckets is fully utilized to achieve lightweight data display and on-demand loading. This significantly reduces the system's I / O pressure and network bandwidth consumption, improves overall response performance and user experience, and is particularly suitable for high-concurrency, high-data-volume cloud storage scenarios.

[0064] In some embodiments, the bucket allocation strategy is further used to specify a first data volume threshold, and after saving the metadata of the first object to the first directory bucket, it also includes, but is not limited to, the following steps a1 to a3:

[0065] Step a1: Determine whether the metadata in the first directory bucket exceeds the first data volume threshold.

[0066] Specifically, when setting the bucket directory strategy, the user also sets a first data volume threshold, which limits the maximum amount of metadata stored in the directory bucket (i.e., the upper limit of the metadata storage in the directory bucket). Based on the first data volume upper limit, the electronic device of this application method can periodically check whether the amount of metadata in the first directory bucket exceeds the first data volume threshold.

[0067] Step a2: If the amount of metadata in the first directory bucket exceeds the first data volume threshold, then multiple subdirectory buckets are created, and the metadata in the first directory bucket is divided into multiple subsets, with each subset corresponding to a subdirectory bucket.

[0068] Specifically, when the real-time metadata storage volume of the first directory bucket exceeds a first data volume threshold, an expansion operation is performed. Based on the bucket allocation strategy, the electronic device of this application creates multiple sub-directory buckets and divides the metadata in the first directory bucket into multiple subsets. The data volume of the sub-directory buckets and the number of subsets are kept consistent.

[0069] In this embodiment, the creation of subdirectory buckets can take into account the following factors: current system load, data distribution characteristics, and expected data growth trends. When partitioning the metadata of the first directory bucket, multiple subsets can be formed based on a specific algorithm (such as hash partitioning, range partitioning, or list partitioning). Each subset represents a portion of the original metadata and corresponds to a newly created subdirectory bucket.

[0070] Step a3: Migrate each subset of data in the first directory bucket to the corresponding subdirectory bucket.

[0071] Specifically, after the subsets are divided, a data migration operation is performed. Each subset is extracted from the first directory bucket and migrated to its corresponding subdirectory bucket. During the migration process, transaction processing mechanisms can be used to ensure data integrity, and data transfer protocols and parallel processing techniques can be employed to accelerate the data migration process.

[0072] In the above embodiments, based on a preset first data volume threshold, a system monitoring mechanism ensures that when the amount of metadata in a directory bucket approaches its capacity limit, it can be detected in a timely manner, and the metadata is split and stored in the corresponding subdirectory buckets. This distributes storage pressure, thereby improving access speed while ensuring the logical coherence between data.

[0073] In some embodiments, when obtaining the metadata of the second object in the second directory bucket in step S204 above, it includes, but is not limited to, the following steps b1 to b2:

[0074] Step b1: If the second directory bucket and the first directory bucket are the same directory bucket, then during the process of migrating each subset of data in the first directory bucket to the subdirectory bucket, the acquisition of the metadata of the second object is paused.

[0075] Specifically, since users manage all data in the storage system through the main bucket and cannot directly see the real-time status of the buckets, when performing data enumeration operations, the metadata in a certain bucket may have reached the first data volume threshold, and a subset may be being partitioned or migrated. To avoid data inconsistencies, duplications, or omissions during data enumeration, the retrieval of metadata for that bucket is paused when data in the process of metadata partitioning or migration is encountered.

[0076] In this embodiment, pausing the acquisition of metadata for the second object may include, but is not limited to, returning a retry reminder to the metadata enumeration request, adding the request to the waiting queue for automatic response after the migration is completed, sending a warning to the management end, or recording it using logs.

[0077] Step b2: After migrating each subset of data in the first directory bucket to the subdirectory bucket, obtain the metadata of the second object from the subdirectory bucket.

[0078] Specifically, once the system detects that all subsets of data in the first bucket have been successfully migrated to their corresponding buckets and that index updates and status checks have been completed, subsequent metadata enumeration requests for the second object will be redirected to the corresponding bucket.

[0079] In this embodiment, after updating the mapping relationship between the storage object and the directory bucket according to the bucket allocation strategy, the metadata of the second object is read from the subdirectory bucket. For example, if the first directory bucket is "bucket", it is split into two subdirectory buckets, "bucket-A" and "bucket-B", according to the hash range. Then, the metadata of the second object is recalculated according to the key value of the second object, and the metadata of the second object is migrated to the two subdirectory buckets respectively.

[0080] In the above embodiments, by pausing acquisition and redirection mechanisms, it is ensured that no concurrent read / write conflicts occur during metadata migration when performing data enumeration operations, thus guaranteeing the integrity of the data enumeration and data migration process.

[0081] In some embodiments, when obtaining the metadata of the second object in the second directory bucket in step S204 above, the method further includes:

[0082] If the second object indicated by the data enumeration request includes multiple objects, and the second directory buckets corresponding to the multiple objects include the first directory bucket and the third directory bucket, then during the process of migrating each subset of data set in the first directory bucket to the subdirectory bucket, the acquisition of metadata from the first directory bucket is paused, and metadata is acquired from the third directory bucket.

[0083] Specifically, during data enumeration, the second object indicated by the data enumeration request may include multiple target storage objects, and the metadata of these target storage objects is stored in different directory buckets. In the above scenario, the electronic device of this application parses the data enumeration request and obtains the target directory buckets involved in the metadata of the target storage objects. If the metadata of some target storage objects is stored in the first directory bucket (i.e., the target directory bucket in the process of subset partitioning or subset migration), the acquisition of the metadata of the target storage objects from the first directory bucket is suspended, and the metadata of the remaining target storage objects is acquired preferentially from the third directory bucket (i.e., the target directory bucket that is not in the process of subset partitioning or subset migration).

[0084] In this embodiment, the method for obtaining the metadata of the target storage objects in the first directory bucket may include, but is not limited to, temporarily mounting the metadata request of the target storage objects in the first directory bucket, waiting for the metadata of the target storage objects in the third directory bucket to complete the enumeration request, and if the first directory bucket is still performing subset partitioning or subset migration, then the metadata of the target storage objects in the third directory bucket is returned, and a data enumeration anomaly is displayed for the metadata of the unenumerated target storage objects; if the subset migration of the first directory bucket has been completed, then the metadata of the target storage objects in the first directory bucket is enumerated, and the data is returned in combination with the metadata of the target storage objects in the third directory bucket.

[0085] In the above embodiments, when dealing with metadata distributed in different directory buckets, temporary mounting and priority reading operations are used to avoid the entire data enumeration request being blocked due to the migration of a single directory bucket, thereby significantly improving the system's concurrent processing capability.

[0086] In some embodiments, after each subset of data in the first directory bucket is migrated to its corresponding subdirectory bucket, the method further includes:

[0087] Release the storage space corresponding to the first directory bucket, or set the storage space corresponding to the first directory bucket to hibernation mode.

[0088] Specifically, once it is confirmed that each subset of the metadata in the first directory bucket has been migrated to its corresponding subdirectory bucket, and the metadata-related attribute information (such as index information, routing tables, etc.) has been updated to the subdirectory bucket structure, the system determines that there is no data in the first directory bucket and can proceed with the resource reclamation process.

[0089] In this embodiment, for the first directory bucket that can enter the resource reclamation process, the following two operations can be performed on the directory bucket. First, completely release the storage space corresponding to the first directory bucket, including but not limited to clearing the metadata records stored in the first directory bucket and releasing the occupied storage resources; Second, considering historical access needs or possible future backtracking queries, the system does not immediately release the storage space of the first directory bucket, but sets it to a dormant state. In the dormant state, the first directory bucket no longer receives data storage and data enumeration functions.

[0090] In the above embodiments, by releasing storage space or implementing hibernation management for the first directory bucket after migration, the reclamation and lifecycle management of metadata storage resources are achieved. This avoids some buckets occupying system resources for extended periods, significantly improving the operating efficiency of the storage system.

[0091] In some embodiments, the bucket allocation strategy is further configured to specify a second data volume threshold, which is less than a first data volume threshold. After migrating each subset of data in the first directory bucket to its corresponding subdirectory bucket, the strategy further includes:

[0092] If, among the multiple sub-directory buckets corresponding to the first directory bucket, there exists a sub-directory bucket with a data volume lower than the second data volume threshold, and the sum of the data volumes of the multiple sub-directory buckets is not greater than the first data volume threshold, then the metadata of the multiple sub-directory buckets will be merged into the same directory bucket.

[0093] Specifically, since the storage volume of objects within a directory bucket is not constant, in addition to the writing of new storage objects, the data volume of storage objects within the directory bucket may also decrease due to factors such as the lifecycle of existing storage objects. When setting the bucket allocation strategy, the user simultaneously sets a second data volume threshold. This second data volume threshold is used to limit the minimum storage volume of metadata in the subdirectory bucket (i.e., the lower limit of the metadata storage volume of the subdirectory bucket), and this second data volume threshold is always less than the aforementioned first data volume threshold. Based on the second data volume threshold, the electronic device of this application method can periodically check whether the amount of metadata in the second directory bucket is lower than the second data volume threshold.

[0094] In this embodiment, when a first subdirectory bucket has a data volume lower than the second data volume threshold, the status of its parent bucket is checked. If the parent bucket has already released its storage space, it is checked whether there are other second subdirectory buckets with data volumes lower than the second data volume threshold and whose parent buckets have already released their storage space. If such a second subdirectory bucket exists, it is determined whether the sum of the data volumes of the first and second subdirectory buckets is less than the first data volume threshold. If the condition is met, the first and second subdirectory buckets are merged.

[0095] In this embodiment, when there is a subdirectory bucket with a data volume lower than the second data volume threshold, the status of the parent directory bucket of the subdirectory bucket is checked. If the parent directory bucket is set to a dormant state, the other subdirectory buckets corresponding to the metadata in the original parent directory bucket are checked. It is determined whether the sum of the data volume of multiple subdirectory buckets is less than the first data volume threshold. If the condition is met, the multiple subdirectory buckets are merged into the original parent directory bucket.

[0096] In this embodiment, subdirectory merging can also be based on a comprehensive judgment of factors such as the access frequency of stored objects and business cycles. For example, low-capacity subdirectory buckets that have not been accessed for a long time are selected for priority merging.

[0097] In the above embodiments, the second data volume threshold of the subdirectory bucket is used to avoid excessive data in the subdirectory bucket due to excessive splitting of the directory bucket. Based on the reverse merging mechanism, it supports both directory bucket splitting under high load and subdirectory bucket merging under low load.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0099] Corresponding to the data storage method, this application also provides a data storage device. (See also...) Figure 4 This is a schematic diagram of a spread spectrum clock signal simulation device provided in some embodiments of this application. Figure 4 In this context, the data storage device includes:

[0100] The instruction receiving module 401 is used to receive bucket creation instructions and bucket allocation strategies.

[0101] The bucket creation module 402 is used to create a main bucket and at least one directory bucket according to the bucket allocation strategy. The bucket allocation strategy is used to specify the objects corresponding to each directory bucket. The directory bucket is used to store the metadata of the corresponding objects, and the main bucket is used to store the entity data of each object.

[0102] The data storage module 403 is used to respond to receiving a storage instruction for the first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket.

[0103] In some alternative implementations, the data storage device may further include a data enumeration module, a data search module, and a data display module.

[0104] The data enumeration module is used to receive data enumeration requests, which indicate the second object to be displayed.

[0105] The data acquisition module is used to locate the second directory bucket corresponding to the second object and retrieve the metadata of the second object from the second directory bucket.

[0106] The data display module is used to display the second object based on its metadata.

[0107] In some alternative implementations, the data display module includes:

[0108] The attribute data unit is used to retrieve the attribute data of the second object from the metadata of the second object and to display the attribute data.

[0109] A storage address unit is used to obtain the entity data storage address of the second object from the metadata of the second object in response to the selection of the attribute data of the second object.

[0110] The entity data unit is used to retrieve and display the entity data of the second object based on the entity data storage address.

[0111] In some optional implementations, the bucket allocation strategy is also used to specify a first data volume threshold, and the data storage module 403 includes:

[0112] The threshold judgment unit is used to determine whether the amount of metadata in the first directory bucket exceeds the first data amount threshold.

[0113] The subset partitioning unit is used to create multiple sub-directory buckets if the amount of metadata in the first directory bucket exceeds the first data volume threshold, and to divide the metadata in the first directory bucket into multiple subsets, with each subset corresponding to a sub-directory bucket.

[0114] The sub-dataset migration unit is used to migrate each sub-dataset in the first directory bucket to its corresponding sub-directory bucket.

[0115] In some alternative implementations, the data acquisition module includes:

[0116] The pause acquisition unit is used to pause the acquisition of metadata of the second object during the process of migrating each subset of data in the first directory bucket to the subdirectory bucket if the second directory bucket and the first directory bucket are the same directory bucket.

[0117] The first metadata acquisition unit is used to acquire the metadata of the second object from the sub-directory bucket after migrating each subset of the dataset in the first directory bucket to the sub-directory bucket.

[0118] In some optional implementations, the data acquisition module further includes:

[0119] The second metadata acquisition unit is used to pause the acquisition of metadata from the first directory bucket and acquire metadata from the third directory bucket during the process of migrating each subset of data set in the first directory bucket to the subdirectory bucket if the second object indicated by the data enumeration request includes multiple objects and the second directory buckets corresponding to the multiple objects include the first directory bucket and the third directory bucket.

[0120] In some optional implementations, after migrating each subset of data in the first directory bucket to its corresponding subdirectory bucket, the data storage method further includes:

[0121] The directory bucket status setting module is used to release the storage space corresponding to the first directory bucket, or to set the storage space corresponding to the first directory bucket to a dormant state.

[0122] In some optional implementations, the bucket allocation strategy is further used to specify a second data volume threshold, which is less than a first data volume threshold. After migrating each subset of data in the first directory bucket to its corresponding subdirectory bucket, the data storage method further includes:

[0123] If, in the multiple sub-directory buckets corresponding to the first directory bucket, there is a sub-directory bucket with a data volume lower than the second data volume threshold, and the sum of the data volumes of the multiple sub-directory buckets is not greater than the first data volume threshold, then the metadata of the multiple sub-directory buckets will be merged into the same directory bucket.

[0124] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0125] See also Figure 5 Embodiments of this application also provide an electronic device including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data storage method embodiments.

[0126] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage method embodiments when it is run.

[0127] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0128] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0129] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0130] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] The data storage method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data storage method, characterized in that, The method includes: Receive bucket creation instructions and bucket allocation policies; According to the bucket allocation strategy, a primary bucket and at least one directory bucket are created, wherein the bucket allocation strategy is used to specify the objects corresponding to each directory bucket, the directory bucket is used to store the metadata of the corresponding objects, and the primary bucket is used to store the entity data of each object; In response to receiving a storage instruction for a first object, the system locates the first directory bucket corresponding to the first object, saves the metadata of the first object to the first directory bucket, and saves the entity data of the first object to the main bucket. The method further includes: Receive a data enumeration request, the data enumeration request being used to indicate a second object to be displayed; Locate the second directory bucket corresponding to the second object, and retrieve the metadata of the second object from the second directory bucket; Based on the metadata of the second object, display the second object; The process of retrieving the metadata of the second object from the second directory bucket includes: If the second object indicated by the data enumeration request includes multiple objects, and the second directory bucket corresponding to the multiple objects includes the first directory bucket and the third directory bucket, then during the process of migrating each subset of data set in the first directory bucket to the subdirectory bucket, the acquisition of metadata from the first directory bucket is paused, and metadata is acquired from the third directory bucket. The bucket allocation strategy is further used to specify a first data volume threshold. After saving the metadata of the first object to the first directory bucket, the method further includes: Determine whether the amount of metadata in the first directory bucket exceeds the first data volume threshold; If the amount of metadata in the first directory bucket exceeds the first data volume threshold, then multiple sub-directory buckets are created, and the metadata in the first directory bucket is divided into multiple subsets, with each subset corresponding to a sub-directory bucket. Migrate each subset of data in the first directory bucket to its corresponding subdirectory bucket; The bucket allocation strategy further specifies a second data volume threshold, which is less than the first data volume threshold. After migrating each subset of data in the first directory bucket to its corresponding subdirectory bucket, the method further includes: If, among the multiple sub-directory buckets corresponding to the first directory bucket, there exists a sub-directory bucket with a data volume lower than the second data volume threshold, and the sum of the data volumes of the multiple sub-directory buckets is not greater than the first data volume threshold, then the metadata of the multiple sub-directory buckets will be merged into the same directory bucket.

2. The method according to claim 1, characterized in that, The step of displaying the second object based on its metadata includes: Obtain the attribute data of the second object from the metadata of the second object, and display the attribute data; In response to the selection of the attribute data of the second object, the entity data storage address of the second object is obtained from the metadata of the second object; Based on the entity data storage address, obtain and display the entity data of the second object.

3. The method according to claim 1, characterized in that, The step of obtaining the metadata of the second object from the second directory bucket includes: If the second directory bucket and the first directory bucket are the same directory bucket, then during the process of migrating each subset of data in the first directory bucket to the subdirectory bucket, the acquisition of the metadata of the second object is paused; After migrating each subset of data set in the first directory bucket to the subdirectory bucket, the metadata of the second object is obtained from the subdirectory bucket.

4. The method according to claim 1, characterized in that, After migrating each subset of data in the first directory bucket to its corresponding subdirectory bucket, the method further includes: Release the storage space corresponding to the first directory bucket, or set the storage space corresponding to the first directory bucket to a hibernation state.

5. A data storage device, characterized in that, The device includes: The instruction receiving module is used to receive bucket creation instructions and bucket allocation policies; The bucket creation module is used to create a main bucket and at least one directory bucket according to the bucket allocation strategy, wherein the bucket allocation strategy is used to specify the objects corresponding to each directory bucket, the directory bucket is used to store the metadata of the corresponding objects, and the main bucket is used to store the entity data of each object; The data storage module is configured to, in response to receiving a storage instruction for a first object, locate the first directory bucket corresponding to the first object, save the metadata of the first object to the first directory bucket, and save the entity data of the first object to the main bucket; The device further includes: The data enumeration module is used to receive a data enumeration request, wherein the data enumeration request is used to indicate a second object to be displayed; The data acquisition module is used to locate the second directory bucket corresponding to the second object, and to obtain the metadata of the second object from the second directory bucket; The data display module is used to display the second object based on its metadata. The data acquisition module includes: The second metadata acquisition unit is configured to, if the second object indicated by the data enumeration request includes multiple objects, and the second directory bucket corresponding to the multiple objects includes the first directory bucket and the third directory bucket, then during the process of migrating each subset of data set in the first directory bucket to the subdirectory bucket, pause the acquisition of metadata from the first directory bucket and acquire metadata from the third directory bucket. The device further includes: A threshold determination unit is used to determine whether the amount of metadata in the first directory bucket exceeds a first data amount threshold. A subset partitioning unit is used to create multiple sub-directory buckets and divide the metadata in the first directory bucket into multiple subsets if the amount of metadata in the first directory bucket exceeds the first data volume threshold. The subsets correspond one-to-one with the sub-directory buckets. The bucket allocation strategy is also used to specify the first data volume threshold. The sub-dataset migration unit is used to migrate each sub-dataset in the first directory bucket to its corresponding sub-directory bucket; The device further includes: The data merging module is used to merge the metadata of the multiple subdirectory buckets into the same directory bucket if, among the multiple subdirectory buckets corresponding to the first directory bucket, there is a subdirectory bucket with a data volume lower than a second data volume threshold, and the sum of the data volumes of the multiple subdirectory buckets is not greater than the first data volume threshold; wherein, the bucket allocation strategy is further used to specify a second data volume threshold, the second data volume threshold being less than the first data volume threshold.

6. An electronic device, characterized in that, include: Memory: Used to store computer programs; A processor for implementing the steps of the data storage method as described in any one of claims 1 to 4 when executing the computer program.

Images