Data storage method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data storage method, device, electronic equipment and storage medium, particularly relates to the field of artificial intelligence such as object storage, distributed storage and cloud computing, and the method can comprise: acquiring a data shard to be processed, generating an object in a public cloud object storage system for the data shard, the data shard comprising: a data shard in a file of a user that needs to be migrated from a network disk file storage system to the public cloud object storage system; determining a storage bucket to which the object belongs, and generating identification information of the object; storing the identification information of the storage bucket and the identification information of the object into meta information, the meta information being meta information in a data object meta information management layer in a network disk file storage mode, and storing the data shard by using the object. By applying the scheme, the implementation complexity and cost can be reduced, and the processing efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to data storage methods, apparatus, electronic devices and storage media in the fields of object storage, distributed storage and cloud computing. Background Technology

[0002] Cloud storage services are personal cloud storage services, and their file storage method differs from that of common public cloud services from the outset. In multi-cloud scenarios, there is a need to migrate some users' file data to public cloud services, specifically to public cloud object storage systems. Current implementations for this migration are typically complex and inefficient. Summary of the Invention

[0003] This disclosure provides data storage methods, apparatus, electronic devices, and storage media.

[0004] A data storage method, comprising:

[0005] Obtain the data shards to be processed, and generate objects in the public cloud object storage system for the data shards. The data shards include: data shards in the files of users who need to be migrated from the cloud drive file storage system to the public cloud object storage system.

[0006] Determine the storage bucket to which the object belongs, and generate the object's identification information;

[0007] The identification information of the storage bucket and the identification information of the object are stored in the metadata, which is the metadata in the data object metadata management layer of the cloud disk file storage method, and the data fragments are stored using the object.

[0008] A data storage device includes: an object generation module, an information acquisition module, and a storage module;

[0009] The object generation module is used to obtain data fragments to be processed and generate objects in the public cloud object storage system for the data fragments. The data fragments include data fragments in the files of users who need to be migrated from the cloud drive file storage system to the public cloud object storage system.

[0010] The information acquisition module is used to determine the storage bucket to which the object belongs and to generate the object's identification information;

[0011] The storage module is used to store the identification information of the storage bucket and the identification information of the object into the metadata, wherein the metadata is the metadata in the data object metadata management layer of the cloud disk file storage method, and the object is used to store the data fragment.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart illustrating an embodiment of the data storage method described in this disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the relationship between the file metadata management layer, the data object metadata management layer, and the data fragmentation storage layer as described in this disclosure;

[0022] Figure 3 This is a schematic diagram of the composition structure of embodiment 300 of the data storage device described in this disclosure;

[0023] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0026] Figure 1 This is a flowchart illustrating an embodiment of the data storage method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0027] In step 101, the data shards to be processed are obtained, and objects in the public cloud object storage system are generated for the data shards. The data shards include data shards in the files of users who need to migrate from the cloud drive file storage system to the public cloud object storage system.

[0028] In step 102, the bucket to which the object belongs is determined, and the object's identification information is generated.

[0029] In step 103, the identification information of the storage bucket and the identification information of the object are stored in the metadata. The metadata is the metadata in the data object metadata management layer of the cloud disk file storage method, and the data fragment is stored using the object.

[0030] As can be seen, by adopting the solution described in the above method embodiments, the migration of user file data from the cloud storage system to the public cloud object storage system can be realized without changing the existing storage architecture. The migration development difficulty is relatively small, thereby reducing the implementation complexity and cost, and improving the processing efficiency. Moreover, multiple data fragments of a file can be mapped to different objects in the public cloud object storage system, thereby maximizing the support for concurrent upload and download of a single file and improving upload and download speeds.

[0031] The data shards to be processed can refer to data shards in a user's files that need to be migrated from a cloud storage system to a public cloud object storage system. In practice, migration is usually performed at the user level. That is, if a user's file data needs to be migrated, their existing data will be migrated, and new data will be written directly to the system. Accordingly, the data shards to be processed can refer to either the user's existing data or new data.

[0032] Additionally, it should be noted that the migration can be carried out with the user's authorization, that is, with the user's authorization, the user's file data can be migrated from the cloud storage system to the public cloud object storage system.

[0033] According to the current implementation, the file storage method of cloud storage is mainly divided into three layers: the file metadata management layer, the data object metadata management layer, and the data sharding storage layer, which will be introduced below.

[0034] 1) File Metadata Management Layer: Responsible for storing file metadata, which may include user identifier, file identifier, file path and file name, file creation / modification / viewing time, file type and file size, etc.

[0035] 2) Data Object Metadata Management Layer: Responsible for storing the metadata of the binary data corresponding to the file, which may include data fingerprints, data fragmentation strategies, and data routing information. The same file from different users (such as files transferred instantly or copied files) may correspond to the same data object.

[0036] Data fingerprints may include Message Digest Algorithm 5 (MD5), Secure Hash Algorithm 1 (SHA1), and Cyclic Redundancy Check 32 (CRC32), etc.

[0037] The data sharding strategy records the size of the data shards that a file can be split into, which is usually 4*2^n MB (n>=0, 4MB, 8MB, 16MB, 32MB, …).

[0038] The data routing information includes information such as the cluster where the data is located and the slice key of each data slice. The cluster can be named clusterX, supports unlimited horizontal scaling, and is the foundation for the cloud drive to support large-scale data storage. Different data slices of the same file are stored in the same cluster.

[0039] 3) Data Sharding Storage Layer: A distributed key-value (KV) storage system can be used to store the actual data content of files. The data content of a file (data object) can be split into one or more data shards according to a certain strategy. Furthermore, the sliceKey can be generated by the KV storage system or generated by the business logic and then passed to the KV storage system.

[0040] Accordingly, Figure 2 This is a schematic diagram illustrating the relationship between the file metadata management layer, the data object metadata management layer, and the data fragmentation storage layer described in this disclosure.

[0041] Generally speaking, it is desirable to keep the user file operation interface unchanged. Correspondingly, the file metadata management layer needs to remain unchanged. Therefore, taking into account the compatibility with existing systems and the flexibility of data management, the solution described in this disclosure is proposed, which is to use a public cloud object storage system to replace the KV storage system, and store each data shard as an object in the public cloud object storage system.

[0042] Specifically, for the acquired data fragments to be processed (the file can be split into multiple data fragments of size 4*2^n MB in sequence, except for the last data fragment), an object in a public cloud object storage system can be generated for the data fragments to be processed, and the bucket to which the object belongs and the identification information of the generated object can be determined.

[0043] Preferably, the bucket to which the object belongs can be determined based on the file type corresponding to the data fragment to be processed and the acquisition time of the data fragment to be processed, and the file type is the file type of the file to which the data fragment to be processed belongs.

[0044] The above method can efficiently and accurately determine the bucket to which an object belongs, thus laying a good foundation for subsequent processing.

[0045] Preferably, in response to determining that there exists a bucket that matches the file type corresponding to the data fragment to be processed and the acquisition time of the data fragment to be processed, the matching bucket is taken as the bucket to which the object belongs. The matching bucket includes: the corresponding file type is the file type corresponding to the data fragment to be processed, and the corresponding time period includes the acquisition time of the data fragment to be processed. Otherwise, a new bucket is created, and the created bucket is taken as the bucket to which the object belongs. Each bucket corresponds to a file type and a time period, and the time periods corresponding to any two buckets corresponding to the same file type do not overlap.

[0046] Cloud storage user files typically exhibit clear time characteristics; files uploaded a long time ago are usually accessed less frequently. Therefore, setting (i.e. creating) buckets based on time is meaningful. In addition, file types can be broadly categorized into images, videos, audio, documents, and compressed files, and the consumption behavior for different types of files varies greatly. Therefore, setting buckets based on file type is also meaningful.

[0047] Accordingly, the solution described in this disclosure can adopt a bucketing strategy based on file type and time, thereby making it easy to utilize the statistical capabilities of public cloud object storage, such as performing some access behavior-related statistical analysis, helping to analyze the data access characteristics of different types of files, and laying a solid foundation for future hierarchical data storage.

[0048] Preferably, the time periods corresponding to different buckets of the same file type are of the same duration, which further facilitates the statistical analysis, etc.

[0049] In practical applications, when the data volume is small, a bucket can be set up for each file type every month or week. For example, one bucket could correspond to January 2022, another bucket to February 2022, and so on. As the data volume increases, a bucket can be set up for each file type every day, or multiple buckets can be set up for each day, such as one bucket for each hour of the day. For example, for January 1, 2022, one bucket could correspond to 0:00 to 1:00, another bucket to 1:00 to 2:00, and so on. In this way, for the data fragments to be processed, the corresponding bucket can be determined based on the file type of the data fragment and the acquisition time of the data fragment.

[0050] Preferably, the method for generating the object's identification information may include: obtaining the key of the file to which the data slice to be processed belongs, and obtaining the slice number of the data slice to be processed, wherein the slice number is determined based on the position of the data slice to be processed in the file to which it belongs; generating the sliceKey of the data slice to be processed by hashing based on the file's key and the slice number; and using the generated sliceKey as the object's identification information.

[0051] Thus, sliceKey = hash(uniqueKey, sliceNum); (1)

[0052] Here, uniqueKey represents the file key, and sliceNum represents the slice number. For example, in order, the slice number of the first data slice can be 0, the slice number of the second data slice can be 1, the slice number of the third data slice can be 2, and so on.

[0053] As mentioned earlier, sliceKey can be generated by the KV storage system or by the business. The method shown in formula (1) is the method of being generated by the business. This method is simple to implement and can reduce the storage space occupied compared to the method of being generated by the KV storage system and requiring the storage of a list of sliceKeys for all data slices.

[0054] After determining the bucket to which the object belongs and generating the object's sliceKey, the bucket's identification information and sliceKey can be stored in the metadata. The metadata is the metadata in the data object metadata management layer of the cloud drive file storage method, and the object can be used to store the data fragments to be processed.

[0055] Preferably, the bucket's identification information can be stored in the cluster number field of the metadata, and the object's sliceKey can be stored in the sliceKey field of the metadata.

[0056] The existing cluster and sliceKey fields in the metadata can be used to store the bucket identification information and the object's sliceKey, respectively. In public cloud object storage systems, a file object is usually marked in the form of a bucket-object. The sliceKey of the object described in this disclosure is equivalent to the object. In this way, there is no need to modify the existing fields of the metadata, that is, there is no need to add additional fields to store the bucket-object information in the public cloud object storage system, thereby further reducing the implementation complexity and implementation cost.

[0057] In addition, preferably, the public cloud object storage system in the scheme described in this disclosure can be a Simple Storage Service (S3) public cloud object storage system.

[0058] S3 is a scalable cloud storage service, also known as bucket storage. It is an internet-based storage service that can provide temporary or permanent storage services for any type of file. It has many advantages such as security, reliability, ease of use, and low cost.

[0059] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0060] In summary, the solution described in this disclosure does not require changes to the existing storage architecture, making migration and development less difficult, thereby reducing implementation complexity and costs, improving processing efficiency, and easily supporting existing data management capabilities such as instant upload, deduplication, and data recycling. In addition, multiple data shards of a file can be mapped to different objects in the public cloud object storage system, thereby maximizing support for concurrent uploads and downloads of single files and improving upload and download speeds. Furthermore, a file type + time bucketing strategy can be adopted, which can easily utilize the statistical capabilities of the public cloud object storage.

[0061] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0062] Figure 3 This is a schematic diagram of the structural composition of embodiment 300 of the data storage device described in this disclosure. Figure 3 As shown, it includes: an object generation module 301, an information acquisition module 302, and a storage module 303.

[0063] The object generation module 301 is used to obtain the data fragments to be processed and generate objects in the public cloud object storage system for the data fragments. The data fragments include data fragments in the files of users who need to migrate from the cloud disk file storage system to the public cloud object storage system.

[0064] The information acquisition module 302 is used to determine the bucket to which the object belongs and to generate the object's identification information.

[0065] Storage module 303 is used to store the bucket identification information and the object identification information into the metadata, wherein the metadata is the metadata in the data object metadata management layer in the cloud disk file storage method, and the data fragments are stored using the object.

[0066] Using the solution described in the above device embodiment, user file data can be migrated from a cloud storage system to a public cloud object storage system without changing the existing storage architecture. The migration development is relatively simple, thereby reducing implementation complexity and cost, and improving processing efficiency. Moreover, multiple data fragments of a file can be mapped to different objects in the public cloud object storage system, thereby maximizing support for concurrent uploads and downloads of single files and improving upload and download speeds.

[0067] Data shards to be processed can refer to data shards in user files that need to be migrated from a cloud storage system to a public cloud object storage system.

[0068] According to the current implementation, the file storage method of cloud storage is mainly divided into three layers: the file metadata management layer, the data object metadata management layer, and the data sharding storage layer.

[0069] Generally speaking, it is desirable to keep the user file operation interface unchanged. Correspondingly, the file metadata management layer needs to remain unchanged. Therefore, considering the compatibility with existing systems and the flexibility of data management, the solution described in this disclosure is proposed, which uses a public cloud object storage system to replace the KV storage system in the data shard storage layer, and each data shard is stored as an object in a public cloud object storage system.

[0070] Specifically, for the acquired data fragments to be processed, the object generation module 301 can generate an object in a public cloud object storage system for it, and the information acquisition module 302 can determine the bucket to which the object belongs, as well as the identification information of the generated object.

[0071] Preferably, the information acquisition module 302 can determine the bucket to which the object belongs based on the file type corresponding to the data fragment to be processed and the acquisition time of the data fragment to be processed, and the file type is the file type of the file to which the data fragment to be processed belongs.

[0072] Preferably, in response to determining that there exists a bucket that matches the file type corresponding to the data segment to be processed and the acquisition time of the data segment to be processed, the information acquisition module 302 takes the matching bucket as the bucket to which the object belongs. The matching bucket includes: the corresponding file type is the file type corresponding to the data segment to be processed, and the corresponding time period includes the acquisition time of the data segment to be processed. Otherwise, a new bucket is created and the created bucket is taken as the bucket to which the object belongs. Each bucket corresponds to a file type and a time period, and the time periods corresponding to any two buckets corresponding to the same file type do not overlap.

[0073] Preferably, the time periods corresponding to different buckets of the same file type have the same duration.

[0074] In practical applications, when the data volume is small, a bucket can be set up for each file type every month or week. For example, one bucket could correspond to January 2022, another bucket to February 2022, and so on. As the data volume increases, a bucket can be set up for each file type every day, or multiple buckets can be set up for each day, such as one bucket for each hour of the day. For example, for January 1, 2022, one bucket could correspond to 0:00 to 1:00, another bucket to 1:00 to 2:00, and so on. In this way, for the data fragments to be processed, the corresponding bucket can be determined based on the file type of the data fragment and the acquisition time of the data fragment.

[0075] In addition, preferably, the method by which the information acquisition module 302 generates the object's identification information may include: obtaining the key of the file to which the data slice to be processed belongs, and obtaining the slice number of the data slice to be processed, wherein the slice number is determined according to the position of the data slice to be processed in the file to which it belongs; generating the sliceKey of the data slice to be processed by means of a hash method based on the file's key and the slice number; and using the generated sliceKey as the object's identification information.

[0076] After determining the bucket to which the object belongs and generating the object's sliceKey, the storage module 303 can store the bucket's identification information and the object's sliceKey into the metadata. The metadata is the metadata in the data object metadata management layer of the cloud drive file storage method, and the object can be used to store the data fragments to be processed.

[0077] Preferably, the storage module 303 can store the bucket's identification information in the cluster field of the metadata, and can store the object's sliceKey in the sliceKey field of the metadata.

[0078] In addition, preferably, the public cloud object storage system can be an S3 public cloud object storage system.

[0079] Figure 3 The specific workflow of the device embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0080] In summary, the solution described in the embodiments of this disclosure does not require changes to the existing storage architecture, has lower migration and development difficulty, thereby reducing implementation complexity and cost, improving processing efficiency, and conveniently supporting existing data management capabilities such as instant transmission, deduplication, and recycling. In addition, multiple data fragments of a file can be mapped to different objects in the public cloud object storage system, thereby maximizing support for concurrent uploads and downloads of single files and improving upload and download speeds. Furthermore, a file type + time bucketing strategy can be adopted, thereby conveniently utilizing the statistical capabilities of the public cloud object storage.

[0081] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as object storage, distributed storage, and cloud computing. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0082] The documents described in the embodiments of this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0083] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0084] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0085] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0086] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the methods described in this disclosure by any other suitable means (e.g., by means of firmware).

[0088] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0092] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0093] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0094] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0095] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data storage method, comprising: Obtain the data shards to be processed, and generate objects in the public cloud object storage system for the data shards. The data shards include data shards in the files of users who need to migrate from the cloud drive file storage system to the public cloud object storage system. The cloud drive file storage method includes the following three layers: file metadata management layer, data object metadata management layer, and data shard storage layer, and the public cloud object storage system is used as the data shard storage layer. Determine the bucket to which the object belongs, whether the bucket is an existing bucket or a newly created bucket for the object, and generate the object's identification information; Storing the identification information of the storage bucket and the identification information of the object into the metadata includes: storing the identification information of the storage bucket into the cluster number field of the metadata, storing the identification information of the object into the shard key field of the metadata, wherein the metadata is the metadata in the data object metadata management layer, and the object is used to store the data shard.

2. The method according to claim 1, wherein, Determining the bucket to which the object belongs includes: Based on the file type corresponding to the data shard and the acquisition time of the data shard, the storage bucket to which the object belongs is determined, where the file type is the file type of the file to which the data shard belongs.

3. The method according to claim 2, wherein, The step of determining the storage bucket to which the object belongs based on the file type corresponding to the data shard and the acquisition time of the data shard includes: In response to determining that there exists a storage bucket that matches the file type corresponding to the data shard and the acquisition time of the data shard, the matching storage bucket is taken as the storage bucket to which the object belongs. The matching storage bucket includes: the corresponding file type is the file type corresponding to the data shard, and the corresponding time period includes the acquisition time of the data shard. Otherwise, create a new bucket and use the created bucket as the bucket to which the object belongs. Each bucket corresponds to a file type and a time period, and the time periods corresponding to any two buckets with the same file type do not overlap.

4. The method according to claim 3, wherein, The time periods corresponding to different storage buckets of the same file type have the same duration.

5. The method according to claim 1, wherein, The identification information used to generate the object includes: Obtain the key of the file to which the data fragment belongs, and obtain the fragment number of the data fragment, which is determined based on the position of the data fragment in the file to which it belongs; Based on the file key and the fragment number, a fragment key is generated for the data fragment using a hash method, and the fragment key is used as the identification information of the object.

6. The method according to any one of claims 1 to 5, wherein, The public cloud object storage system includes: a simple storage service public cloud object storage system.

7. A data storage device, comprising: Object generation module, information acquisition module, and storage module; The object generation module is used to obtain data fragments to be processed and generate objects in the public cloud object storage system for the data fragments. The data fragments include data fragments in the files of users who need to migrate from the cloud drive file storage system to the public cloud object storage system. The cloud drive file storage method includes the following three layers: file metadata management layer, data object metadata management layer, and data fragment storage layer, using the public cloud object storage system as the data fragment storage layer. The information acquisition module is used to determine the storage bucket to which the object belongs, whether the storage bucket is an existing storage bucket or a newly created storage bucket for the object, and to generate the object's identification information; The storage module is used to store the identification information of the storage bucket and the identification information of the object into the metadata, including: storing the identification information of the storage bucket into the cluster number field of the metadata, storing the identification information of the object into the shard key field of the metadata, wherein the metadata is the metadata in the data object metadata management layer, and using the object to store the data shard.

8. The apparatus according to claim 7, wherein, The information acquisition module determines the storage bucket to which the object belongs based on the file type corresponding to the data shard and the acquisition time of the data shard, wherein the file type is the file type of the file to which the data shard belongs.

9. The apparatus according to claim 8, wherein, The information acquisition module responds to determining that there exists a storage bucket that matches the file type corresponding to the data shard and the acquisition time of the data shard, and uses the matching storage bucket as the storage bucket to which the object belongs. The matching storage bucket includes: the corresponding file type is the file type corresponding to the data shard, and the corresponding time period includes the acquisition time of the data shard. Otherwise, a new storage bucket is created, and the created storage bucket is used as the storage bucket to which the object belongs. Each storage bucket corresponds to a file type and a time period, and the time periods corresponding to any two storage buckets corresponding to the same file type do not overlap.

10. The apparatus according to claim 9, wherein, The time periods corresponding to different storage buckets of the same file type have the same duration.

11. The apparatus according to claim 7, wherein, The information acquisition module obtains the key of the file to which the data fragment belongs, and obtains the fragment number of the data fragment. The fragment number is determined based on the position of the data fragment in the file to which it belongs. Based on the key of the file and the fragment number, a fragment key of the data fragment is generated by hashing, and the fragment key is used as the identification information of the object.

12. The apparatus according to any one of claims 7 to 11, wherein, The public cloud object storage system includes: a simple storage service public cloud object storage system.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-6.

Images