Data set management method and device, electronic equipment and storage medium

Abstract

The invention provides a data set management method and device, electronic equipment and a storage medium, and belongs to the technical field of big data management. The method comprises the following steps: receiving a hosting request for a data set stored by each cloud server; in response to the hosting request, storing a data set stored by each cloud server to a main storage area; synchronously backing up each data set in the main storage area to a backup storage area; receiving a cross-region synchronization request for transmitting the data set from the first cloud server to a second cloud server for storage; and in response to the cross-region synchronization request, transmitting the data set from the first cloud server to the second cloud server for storage. According to the method, the data sets stored by the cloud servers are all stored in the main storage area, centralized and unified management can be carried out on the data sets of the multiple cloud servers and multiple storage types, the data set management efficiency in the multi-cloud environment can be improved, the data storage safety can be improved in a data backup mode, and the data storage efficiency is improved. And cross-region transmission of the data set among different cloud servers can be realized.

Description

Technical Field

[0001] The present invention relates to the field of big data management technology, and in particular to a data set management method, device, electronic device and storage medium. Background Art

[0002] In the field of large-scale model training, datasets, as a core production factor, are often stored in a dispersed manner across different cloud servers and various storage types. This distributed data storage approach has significant drawbacks: because datasets are stored on different cloud servers, users must log in to each cloud server separately to operate them, making efficient and coordinated management of datasets difficult, resulting in low management efficiency and high costs. In other words, existing technologies suffer from inefficient dataset management in multi-cloud environments. Summary of the Invention

[0003] The present invention provides a data set management method, device, electronic device and storage medium to solve the technical problem of low efficiency of data set management in a multi-cloud environment in the prior art.

[0004] The present invention provides a data set management method, comprising:

[0005] Receive hosting requests for data sets stored on various cloud servers;

[0006] In response to the hosting request, the data sets stored in each of the cloud servers are stored in the main storage area.

[0007] According to a data set management method provided by the present invention, after storing the data sets stored by each cloud server in the main storage area in response to the hosting request, the method further includes:

[0008] Synchronously backing up each data set in the primary storage area to the backup storage area;

[0009] After the data set in the primary storage area is mistakenly deleted, the data set is restored from the backup storage area to the primary storage area.

[0010] According to a data set management method provided by the present invention, in response to the hosting request, storing the data sets stored by each cloud server in the main storage area includes:

[0011] In response to the hosting request, generating a data identifier for each data set; storing each data set and the data identifier for each data set in the main storage area;

[0012] The step of synchronously backing up each data set in the primary storage area to the backup storage area includes:

[0013] Synchronously backing up each data set and the data identifier of each data set in the primary storage area to the backup storage area;

[0014] After the data set in the primary storage area is mistakenly deleted, restoring the data set from the backup storage area to the primary storage area includes:

[0015] After the target data set in the primary storage area is mistakenly deleted, the data set with the same data identifier as the target data set is restored from the backup storage area to the primary storage area.

[0016] According to a data set management method provided by the present invention, after synchronously backing up each data set in the primary storage area to the backup storage area, and before restoring the data set from the backup storage area to the primary storage area after accidentally deleting the data set in the primary storage area, the method further includes:

[0017] Establishing a mapping relationship between the primary storage area and the backup storage area;

[0018] After the data set in the primary storage area is mistakenly deleted, restoring the data set from the backup storage area to the primary storage area includes:

[0019] After the data set in the primary storage area is mistakenly deleted, the data set is restored to the primary storage area from the backup storage area corresponding to the primary storage area in the mapping relationship.

[0020] A data set management method according to the present invention further includes:

[0021] receiving a cross-region synchronization request for transferring a data set from a first cloud server to a second cloud server for storage;

[0022] In response to the cross-region synchronization request, the data set is transferred from the first cloud server to the second cloud server for storage.

[0023] According to a data set management method provided by the present invention, in response to the cross-region synchronization request, transmitting the data set from the first cloud server to the second cloud server for storage includes:

[0024] In response to the cross-region synchronization request, transferring the data set from the first cloud server to the object storage of the second cloud server;

[0025] The data set is transferred from the object storage of the second cloud server to the file storage of the second cloud server.

[0026] According to a data set management method provided by the present invention, after transmitting the data set from the first cloud server to the second cloud server for storage in response to the cross-region synchronization request, the method further includes:

[0027] Generate a storage path for the data set in the second cloud server.

[0028] The present invention also provides a data set management device, comprising:

[0029] A management module for receiving hosting requests for data sets stored on each cloud server;

[0030] The storage module is used to store the data sets stored by each of the cloud servers into the main storage area in response to the hosting request.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described data set management methods is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned data set management methods when executed by a processor.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data set management methods.

[0034] The dataset management method, device, electronic device and storage medium provided by the present invention store the datasets stored by each cloud server in the main storage area, and can centrally and uniformly manage datasets of multiple cloud servers and multiple storage types, thereby improving the efficiency of dataset management in a multi-cloud environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 This is one of the flow charts of the data set management method provided by the present invention.

[0037] Figure 2 This is the second flow chart of the data set management method provided by the present invention.

[0038] Figure 3 This is the third flow chart of the data set management method provided by the present invention.

[0039] Figure 4 This is one of the structural diagrams of the data set management device provided by the present invention.

[0040] Figure 5This is the second structural diagram of the data set management device provided by the present invention.

[0041] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0043] It should be noted that, in the description of the present invention, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements. Terms such as "upper" and "lower" indicate positions or relationships based on those shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation, and are therefore not to be construed as limitations on the present invention. Unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be broadly construed, for example, to mean fixed, removable, or integral; mechanical or electrical; direct or indirect through an intermediary; or internal communication between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] The terms "first," "second," and so forth, used herein are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, allowing embodiments of the present invention to be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first," "second," and so forth generally distinguish objects of a single type, and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the connected objects.

[0045] The following combination Figures 1-6 The present invention describes a data set management method, device, electronic device, and storage medium.

[0046] like Figure 1 As shown, the data set management method provided by the present invention includes steps S1 and S2, and is applied to a data set management system.

[0047] Step S1: receiving a hosting request for a data set stored in each cloud server.

[0048] Specifically, users can submit datasets stored on various cloud servers to the dataset management system and initiate a hosting request. The dataset management system then receives the hosting request. Exemplary cloud servers include Alibaba Cloud, Volcano Cloud, Google Cloud, and others. Each cloud server can have different storage types. The datasets of the present invention can include training sets for large models.

[0049] Step S2: In response to the hosting request, the data sets stored in each cloud server are stored in the main storage area.

[0050] Step S2 can adapt the storage data set according to the storage type of the primary storage area. Exemplarily, the storage type of the primary storage area can be CPFS (Cloud Parallel File System), TOS (Torch Object Storage), OSS (Object Storage Service), vePFS (file storage), etc. For example, if the storage type of the primary storage area is OSS, the data sets stored by each cloud server can be stored in the object storage of the primary storage area.

[0051] For example, if a dataset is stored in Alibaba Cloud's OSS storage and Google Cloud's vePFS storage, the user initiates a dataset hosting request to store both Alibaba Cloud and Volcano Cloud's datasets in the primary storage area.

[0052] As can be seen from the above, the dataset management method of the present invention can store all datasets stored in each cloud server in the main storage area, can centrally manage datasets of multiple cloud servers and multiple storage types, and can improve the efficiency of dataset management in a multi-cloud environment.

[0053] Under certain abnormal circumstances, the data set stored in the primary storage area may be accidentally deleted, affecting the use of the data set. Storing the data set only in the primary storage area has the problem of low data storage security.

[0054] In order to improve the security of data storage, in some embodiments, such as Figure 2As shown, after step S2, the data set management method may further include:

[0055] Step S3, synchronously backing up each data set in the primary storage area to the backup storage area;

[0056] Step S4: after the data set in the primary storage area is mistakenly deleted, the data set is restored from the backup storage area to the primary storage area.

[0057] The backup storage area may also have a different storage type, which may be the same as the storage type of the primary storage area, for the convenience of backup.

[0058] For example, if a user mistakenly deletes data set A stored in the primary storage area, data set A can be restored from the backup storage area to the primary storage area.

[0059] This can improve the security of data storage through data backup.

[0060] Considering that there may be multiple data sets stored in the primary storage area, after a data set in the primary storage area is accidentally deleted, a backup data set of the accidentally deleted data set needs to be determined in the backup storage area in order to restore the data set.

[0061] To this end, in some embodiments, step S2 may further include:

[0062] In response to the hosting request, generate data identifications for each data set; store each data set and the data identifications for each data set in a main storage area;

[0063] Step S3 may further include:

[0064] Synchronously back up each data set and data identifier of each data set in the primary storage area to the backup storage area;

[0065] Step S4 may further include:

[0066] After the target data set in the primary storage area is mistakenly deleted, a data set with the same data identifier as the target data set is restored from the backup storage area to the primary storage area.

[0067] For example, if there are 10 data sets, data identifiers of 10 data sets may be generated as 1, 2, 3, ..., 10 respectively. If the data identifier of the target data set that is mistakenly deleted is 1, the data set with data identifier 1 is restored from the backup storage area to the primary storage area.

[0068] In this way, the data sets in the primary storage area and the backup storage area can be associated one by one through data identification, which makes it easy to determine the backup data set of the accidentally deleted data set in the backup storage area, ensuring that the accidentally deleted data set can be restored.

[0069] Considering that there may be multiple primary storage areas and backup storage areas, after a data set in a primary storage area is accidentally deleted, it is necessary to determine the backup storage area corresponding to the primary storage area to ensure that the data set can be restored.

[0070] To this end, after step S3 and before step S4, the data set management method of the present invention may further include:

[0071] Establish a mapping relationship between the primary storage area and the backup storage area;

[0072] Step S4 may further include:

[0073] After a data set in the primary storage area is mistakenly deleted, the data set is restored to the primary storage area from the backup storage area corresponding to the primary storage area in the mapping relationship.

[0074] For example, if the primary storage area includes primary storage area A, primary storage area B, and primary storage area C, and the dataset in primary storage area A is backed up to backup storage area a, the dataset in primary storage area B is backed up to backup storage area b, and the dataset in primary storage area C is backed up to backup storage area c, then a mapping relationship is established between primary storage area A and backup storage area a, a mapping relationship between primary storage area B and backup storage area b, and a mapping relationship between primary storage area C and backup storage area C. If the dataset in primary storage area A is accidentally deleted, and the backup storage area corresponding to primary storage area A in the mapping relationship is backup storage area a, then the dataset is restored from backup storage area a to primary storage area A.

[0075] In this way, by establishing a mapping relationship between the primary storage area and the backup storage area, the backup storage area corresponding to the primary storage area of ​​the accidentally deleted data set can be determined, ensuring that the data set can be restored.

[0076] In some application scenarios, users need to transfer data sets from one cloud server to another cloud server for storage, and need to realize the transmission of data sets between different cloud servers.

[0077] To this end, in some embodiments, such as Figure 3 As shown, the data set management method of the present invention may further include:

[0078] Step S5, receiving a cross-region synchronization request for transferring a data set from the first cloud server to a second cloud server for storage;

[0079] Step S6: In response to the cross-zone synchronization request, the data set is transferred from the first cloud server to the second cloud server for storage.

[0080] Users can submit cross-region synchronization requests in the dataset management system, which then receives them. For example, if a user needs to transfer a dataset stored in Volcano Cloud to Alibaba Cloud's CPFS storage, they can submit a cross-region synchronization request in the dataset management system to transfer the dataset from Volcano Cloud to Alibaba Cloud storage.

[0081] Furthermore, the optimal transmission method for transmitting the data set from the first cloud server to the second cloud server can be automatically selected based on factors such as the network status and storage load of the first cloud server and the second cloud server, thereby improving transmission efficiency.

[0082] When the present invention transfers a dataset from one cloud server to the file storage of another cloud server, for example, when transferring a dataset stored in Volcano Cloud to the CPFS storage of Alibaba Cloud, the dataset can be directly transferred to the CPFS storage of Alibaba Cloud. However, CPFS storage is file storage, and the file storage speed is slow, resulting in low dataset transmission efficiency.

[0083] In order to improve the transmission speed when transferring a data set from one cloud server to another cloud server for file storage, in some embodiments, step S6 may further include:

[0084] In response to the cross-region synchronization request, transferring the data set from the first cloud server to the object storage of the second cloud server;

[0085] The data set is transferred from the object storage of the second cloud server to the file storage of the second cloud server.

[0086] For example, if the first cloud server is Volcano Cloud and the second cloud server is Alibaba Cloud, the dataset can be transferred from Volcano Cloud to Alibaba Cloud's OSS storage first, and then from Alibaba Cloud's OSS storage to Alibaba Cloud's CPFS storage. Because OSS storage is object storage, which is faster than file storage, it can increase the speed of dataset transfer between different cloud servers. Furthermore, since Alibaba Cloud's OSS storage and Alibaba Cloud's CPFS storage are both owned by Alibaba Cloud, the speed of transferring datasets over the intranet is the fastest.

[0087] This improves the transfer speed of datasets from one cloud server to another through object storage. Furthermore, dataset transfers pass through the object storage intermediary, and after the dataset transfer is complete, the dataset in the object storage must be deleted. This increases storage costs, but dataset transfer costs are higher than storage costs. Improving dataset transfer speeds can reduce data transfer costs, ultimately reducing costs overall.

[0088] In some implementations, after step S6, the dataset management method may further include:

[0089] Generate the storage path of the dataset in the second cloud server.

[0090] This makes it easier for users to directly find the required dataset based on the storage path of the dataset in the second cloud server.

[0091] like Figure 4 As shown, the data set management device provided by the present invention includes:

[0092] A management module for receiving hosting requests for data sets stored on each cloud server;

[0093] The storage module is used to store the data sets stored in each cloud server into the main storage area in response to the hosting request.

[0094] It should be noted that the data set management device provided by the present invention can execute the data set management method of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0095] In some embodiments, the storage module may also be used to:

[0096] Synchronously back up each data set in the primary storage area to the backup storage area;

[0097] After the data set in the primary storage area is accidentally deleted, the data set is restored from the backup storage area to the primary storage area.

[0098] In some embodiments, the storage module may also be used to:

[0099] In response to the hosting request, generate data identifications for each data set; store each data set and the data identifications for each data set in a main storage area;

[0100] Synchronously back up each data set and data identifier of each data set in the primary storage area to the backup storage area;

[0101] After the target data set in the primary storage area is mistakenly deleted, a data set with the same data identifier as the target data set is restored from the backup storage area to the primary storage area.

[0102] In some embodiments, the storage module may also be used to:

[0103] Establish a mapping relationship between the primary storage area and the backup storage area;

[0104] After a data set in the primary storage area is mistakenly deleted, the data set is restored to the primary storage area from the backup storage area corresponding to the primary storage area in the mapping relationship.

[0105] In some implementations, the management module may also be used to:

[0106] receiving a cross-region synchronization request for transferring a data set from a first cloud server to a second cloud server for storage;

[0107] like Figure 5 As shown, the data set management device may further include a transmission module for transmitting the data set from the first cloud server to the second cloud server for storage in response to a cross-region synchronization request.

[0108] In some embodiments, the transmission module may also be used to:

[0109] In response to the cross-region synchronization request, transferring the data set from the first cloud server to the object storage of the second cloud server;

[0110] The data set is transferred from the object storage of the second cloud server to the file storage of the second cloud server.

[0111] In some embodiments, the data set management apparatus may further include a generation module configured to generate a storage path for the data set in the second cloud server.

[0112] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call logic instructions in the memory to execute the data set management method.

[0113] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data set management method provided in the above embodiments.

[0115] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is configured to execute the data set management method provided in the above embodiments when the computer program is executed by a processor.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0117] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data set management method, characterized in that: include: Receive hosting requests for data sets stored on various cloud servers; In response to the hosting request, the data sets stored in each of the cloud servers are stored in the main storage area.

2. The data set management method according to claim 1, characterized in that: After storing the data sets stored by each of the cloud servers in the main storage area in response to the hosting request, the method further includes: Synchronously backing up each data set in the primary storage area to the backup storage area; After the data set in the primary storage area is mistakenly deleted, the data set is restored from the backup storage area to the primary storage area.

3. The data set management method according to claim 2, characterized in that: The step of storing the data sets stored by each of the cloud servers in a primary storage area in response to the hosting request includes: In response to the hosting request, generating a data identifier for each data set; storing each data set and the data identifier for each data set in the main storage area; The step of synchronously backing up each data set in the primary storage area to the backup storage area includes: Synchronously backing up each data set and the data identifier of each data set in the primary storage area to the backup storage area; After the data set in the primary storage area is mistakenly deleted, restoring the data set from the backup storage area to the primary storage area includes: After the target data set in the primary storage area is mistakenly deleted, the data set with the same data identifier as the target data set is restored from the backup storage area to the primary storage area.

4. The data set management method according to claim 2, wherein: After synchronously backing up each data set in the primary storage area to the backup storage area, and before restoring the data set from the backup storage area to the primary storage area after accidentally deleting the data set in the primary storage area, the method further includes: Establishing a mapping relationship between the primary storage area and the backup storage area; After the data set in the primary storage area is mistakenly deleted, restoring the data set from the backup storage area to the primary storage area includes: After the data set in the primary storage area is mistakenly deleted, the data set is restored to the primary storage area from the backup storage area corresponding to the primary storage area in the mapping relationship.

5. The data set management method according to claim 1, wherein: Also includes: receiving a cross-region synchronization request for transferring a data set from a first cloud server to a second cloud server for storage; In response to the cross-region synchronization request, the data set is transferred from the first cloud server to the second cloud server for storage.

6. The data set management method according to claim 5, characterized in that: The step of transmitting the data set from the first cloud server to the second cloud server for storage in response to the cross-region synchronization request includes: In response to the cross-region synchronization request, transferring the data set from the first cloud server to the object storage of the second cloud server; The data set is transferred from the object storage of the second cloud server to the file storage of the second cloud server.

7. The data set management method according to claim 5, characterized in that: After transmitting the data set from the first cloud server to the second cloud server for storage in response to the cross-region synchronization request, the method further includes: Generate a storage path for the data set in the second cloud server.

8. A data set management device, characterized in that: include: A management module for receiving hosting requests for data sets stored on each cloud server; The storage module is used to store the data sets stored by each of the cloud servers into the main storage area in response to the hosting request.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data set management method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data set management method according to any one of claims 1 to 7 is implemented.

Images