Data storage methods, devices, equipment, media and products for data centers

Abstract

The present application provides a data storage method, device, equipment, medium and product for a data center. The method includes: receiving a storage request from a first client, the storage request including multiple files to be stored. The multiple files can be classified to obtain data files and code files, and the data files are stored in a local object storage warehouse, and the pointer files of the code files and data files are stored in a local warehouse. Large files (i.e., data files) can be stored in the cloud, alleviating the storage pressure of the local warehouse.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data storage method, device, equipment, medium and product for a data center. Background Art

[0002] In order to meet the needs of developers for high-quality models and datasets, a data model warehouse can be deployed in the data center. The data model warehouse can be used to store a large number of model and dataset resources. Users can access the data model warehouse to pull or push resources (including datasets and / or models).

[0003] As the number of large files in the data model warehouse increases, the number of submissions corresponding to the modification of these large files continues to increase, which will cause the size of the stored files to expand rapidly, occupy more disk space, and place higher requirements on the storage space of the data model warehouse. Summary of the invention

[0004] The present application provides a data storage method, device, equipment, medium and product for a data center, which is used to solve the problem in the related art that the volume of stored files expands rapidly, occupies more disk space, and has high requirements for the storage space of the data model warehouse.

[0005] In a first aspect, the present application provides a data storage method in a data center, wherein the data center includes a local warehouse and a cloud object storage warehouse corresponding to the local warehouse, including:

[0006] Receiving a storage request from a first client, wherein the storage request is for requesting to store a plurality of files;

[0007] Classify the multiple files to obtain data files and code files;

[0008] The data file is stored in a cloud object storage warehouse corresponding to the local warehouse, and the code file and the pointer file of the data file are stored in the local warehouse, wherein the pointer file is used to indicate the storage address of the data file.

[0009] In this embodiment, a storage request from a first client can be received, and the storage request is used to request the storage of multiple files. Then, the multiple files can be classified to obtain data files and code files, and the data files can be stored in a cloud object storage warehouse corresponding to the local warehouse, and the pointer files of the code files and the data files can be stored in the local warehouse. This allows large files (i.e., data files) to be stored in the cloud, alleviating the storage pressure on the local warehouse.

[0010] Optionally, in the method described above, the multiple files are classified to obtain data files and code files, including:

[0011] Determining the occupied storage capacity of the plurality of files;

[0012] Determine, among the multiple files, a file whose storage capacity is greater than or equal to a preset threshold as the data file;

[0013] A file among the multiple files that occupies a storage capacity smaller than the preset threshold is determined as the code file.

[0014] In this embodiment, multiple files to be stored can be classified according to file size, so that different types of files can be stored in different warehouses to alleviate the storage pressure of the local warehouse.

[0015] Optionally, in the method described above, the data center further comprises a plurality of off-site warehouses and cloud object storage warehouses corresponding to the plurality of off-site warehouses, and the local warehouse communicates with the plurality of off-site warehouses via the Internet;

[0016] The method further comprises:

[0017] Determine a tree structure between the local warehouse and the multiple foreign warehouses, wherein the local warehouse is a main node of the tree structure, and the multiple foreign warehouses are child nodes of the tree structure, and the child nodes include multiple levels of child nodes;

[0018] The data file is synchronously sent to the foreign warehouse corresponding to the next-level child node of the master node in the multi-level child nodes through the cloud object storage warehouse corresponding to the local warehouse based on the tree structure;

[0019] The data file is stored in the corresponding cloud object storage warehouse through the external warehouse corresponding to the next level child node of the main node in the multi-level child nodes.

[0020] In this embodiment, data files can be synchronously stored in other foreign warehouses in a tree structure to improve synchronization efficiency, and the intranet clients corresponding to other foreign warehouses can pull data files from other foreign warehouses, which can reduce the transmission time of pulling data files.

[0021] Optionally, the method as described above further comprises:

[0022] receiving a corresponding acquisition request from a second client, where the second client is an intranet client of a target foreign warehouse among the multiple foreign warehouses;

[0023] If the acquisition request is for requesting a code file, sending the target file corresponding to the acquisition request to the second client;

[0024] If the acquisition request is used to request a data file, the target domain name address is determined and sent to the target foreign warehouse. The target domain name address is used to obtain a pointer file of the target file, and the pointer file of the target file is used to obtain the target file.

[0025] In this embodiment, clients of other external warehouses (such as the second client) can obtain data files and code files through the local warehouse, where the code files can be downloaded directly from the local warehouse, and for data files, the target domain name address can be returned to the client, so that the client can obtain the data files from the corresponding cloud object storage warehouse based on the target domain name address.

[0026] Optionally, the method as described above, determining the target domain name address includes:

[0027] Determine whether the target file corresponding to the acquisition request exists in the target cloud object storage warehouse, the target cloud object storage warehouse being the cloud object storage warehouse corresponding to the target foreign warehouse;

[0028] If so, determining that the target domain name address is a first domain name address, the first domain name address is the domain name address of the target cloud object storage warehouse;

[0029] If not, it is determined that the target domain name address is a second domain name address, and the second domain name address is the domain name address of the cloud object storage warehouse corresponding to the local warehouse.

[0030] In this embodiment, if the target file does not exist in the target cloud object storage warehouse corresponding to the foreign warehouse where the second client is located, the second client can be instructed to obtain the target file from the cloud object storage warehouse corresponding to the local warehouse. If the target file exists, the second client can be instructed to obtain the target file directly from the target cloud object storage warehouse, which can reduce the download and transmission time of the target file.

[0031] Optionally, in the above method, determining whether the target file corresponding to the acquisition request exists in the target cloud object storage repository includes:

[0032] Determining whether synchronization and storage of the target file to the target cloud object storage warehouse is completed;

[0033] If so, determining that the target file exists in the target cloud object storage repository;

[0034] If not, it is determined that the target file does not exist in the target cloud object storage repository.

[0035] In this embodiment, since it takes a long time to synchronize and store the target file, it is possible to determine whether the target file exists in the target cloud object storage warehouse by determining whether the synchronization of the target file is complete, thereby avoiding file download errors caused by the target file not existing in the target cloud object storage warehouse due to failure to synchronize the target file.

[0036] In a second aspect, the present application provides a data storage device in a data center, the data center comprising a local warehouse and a cloud object storage warehouse corresponding to the local warehouse, the device comprising:

[0037] A receiving module, configured to receive a storage request from a first client, wherein the storage request is used to request storage of multiple files;

[0038] A processing module, used for classifying the multiple files to obtain data files and code files;

[0039] A storage module is used to store the data file in a cloud object storage warehouse corresponding to the local warehouse, and to store the code file and the pointer file of the data file in the local warehouse, wherein the pointer file is used to indicate the storage address of the data file.

[0040] Optionally, in the above device, the processing module is specifically used for:

[0041] Determining the occupied storage capacity of the plurality of files;

[0042] Determine, among the multiple files, a file whose storage capacity is greater than or equal to a preset threshold as the data file;

[0043] A file among the multiple files that occupies a storage capacity smaller than the preset threshold is determined as the code file.

[0044] Optionally, in the device as described above, the data center further comprises a plurality of off-site warehouses and cloud object storage warehouses corresponding to the plurality of off-site warehouses, and the local warehouse communicates with the plurality of off-site warehouses via the Internet;

[0045] The device further comprises a synchronization module, wherein the synchronization module is used for:

[0046] Determine a tree structure between the local warehouse and the multiple foreign warehouses, wherein the local warehouse is a main node of the tree structure, and the multiple foreign warehouses are child nodes of the tree structure, and the child nodes include multiple levels of child nodes;

[0047] The data file is synchronously sent to the foreign warehouse corresponding to the next-level child node of the master node in the multi-level child nodes through the cloud object storage warehouse corresponding to the local warehouse based on the tree structure;

[0048] The data file is stored in the corresponding cloud object storage warehouse through the external warehouse corresponding to the next level child node of the main node in the multi-level child nodes.

[0049] Optionally, the device as described above further comprises a download module, and the download module is used to:

[0050] receiving a corresponding acquisition request from a second client, where the second client is an intranet client of a target foreign warehouse among the multiple foreign warehouses;

[0051] If the acquisition request is for requesting a code file, sending the target file corresponding to the acquisition request to the second client;

[0052] If the acquisition request is used to request a data file, the target domain name address is determined and sent to the target foreign warehouse. The target domain name address is used to obtain a pointer file of the target file, and the pointer file of the target file is used to obtain the target file.

[0053] Optionally, in the above device, the download module is specifically used for:

[0054] Determine whether the target file corresponding to the acquisition request exists in the target cloud object storage warehouse, the target cloud object storage warehouse being the cloud object storage warehouse corresponding to the target foreign warehouse;

[0055] If so, determining that the target domain name address is a first domain name address, the first domain name address is the domain name address of the target cloud object storage warehouse;

[0056] If not, it is determined that the target domain name address is a second domain name address, and the second domain name address is the domain name address of the cloud object storage warehouse corresponding to the local warehouse.

[0057] Optionally, in the above device, the download module is specifically used for:

[0058] Determining whether synchronization and storage of the target file to the target cloud object storage warehouse is completed;

[0059] If so, determining that the target file exists in the target cloud object storage repository;

[0060] If not, it is determined that the target file does not exist in the target cloud object storage repository.

[0061] The data storage device of the data center provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be described in detail here.

[0062] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0063] The memory stores computer-executable instructions;

[0064] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, they are used to implement the method described in the first aspect.

[0066] The computer-readable storage medium provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be repeated here.

[0067] In a fifth aspect, the present application provides a computer program product, including a computer program, which is used to implement the method described in the first aspect when executed by a computer.

[0068] The computer program product provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar and will not be described in detail here.

[0069] The data storage method, device, equipment, medium and product of the data center provided by the present application can receive a storage request from a first client, and the storage request includes multiple files to be stored. Multiple files can be classified to obtain data files and code files, and the data files can be stored in a local object storage warehouse, and the pointer files of the code files and data files can be stored in a local warehouse. Large files (i.e., data files) can be stored in the cloud, alleviating the storage pressure of the local warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0071] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0072] Figure 2 A schematic diagram of a data storage method for a data center provided in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of the storage code files and data files of the example of this application;

[0074] Figure 4 A schematic diagram of a flow chart of another data storage method for a data center provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of a flow chart of another data center data storage method provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a tree structure exemplified in this application;

[0077] Figure 7 A schematic diagram of a flow chart of another data center data storage method provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of a flow chart of another data center data storage method provided in an embodiment of the present application;

[0079] Fig. 9 A schematic diagram of a data storage method and a data acquisition method for a data center according to an example of the present application;

[0080] Fig.10 A schematic diagram of the structure of a data storage device in a data center provided in an embodiment of the present application;

[0081] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0082] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0083] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0084] With the rapid development of artificial intelligence and machine learning technologies, developers have a growing demand for high-quality models and data sets. In order to meet this demand, a one-stop open source model service platform has emerged. The platform aims to provide developers with a convenient and efficient model usage experience, enabling them to easily use a variety of high-quality model and data set resources. In addition, these platforms also support cross-regional data model downloads and calculations to meet the needs of users in different regions.

[0085] At present, the open source service platform deploys a data model warehouse in a data center. The data model warehouse can be used to store a large number of model and data set resources. Users can access the data model warehouse to pull or push resources. The data model warehouse can be the central warehouse of the open source service platform. The models and data stored in the data model warehouse of the data center can be synchronized to other distributed data centers through public network broadband and Git mirroring, so that each data center can obtain the latest model and data set resources in a timely manner. Moreover, between multiple data centers, a concurrent synchronization mechanism is used to enable multiple data centers to pull data from the central warehouse at the same time, which speeds up the synchronization.

[0086] Since the data model warehouse stores a large number of models and data set resources, as the number of large files in the data model warehouse increases, the number of submissions corresponding to the modification of these large files continues to increase, which will cause the size of the stored files to expand rapidly, occupying more disk space, and placing higher requirements on the storage space of the data model warehouse.

[0087] Moreover, Git (an open source distributed version control system) is mainly designed to efficiently handle version control of text files. Git's version control mechanism saves the complete history of each file, but for large files (such as pictures, videos or binary files, etc.), especially frequently updated files, Git's performance and storage efficiency will decrease. Regardless of whether the user needs the historical version of the file, the user needs to download every version of the file when downloading the file through his client, which undoubtedly increases the download time and brings more waiting time to the user.

[0088] Therefore, the present application provides a data storage method for a data center, which stores code files in a local warehouse of the data center and stores data files in a cloud object storage warehouse corresponding to the local warehouse. This can realize the storage of large files (i.e., data files) in the cloud, alleviating the storage pressure of the local warehouse.

[0089] Combine the following Figure 1 , the application scenarios to which the embodiments of the present application are applicable are described.

[0090] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the data center 10 includes a local warehouse 101, a cloud object storage warehouse 102 corresponding to the local warehouse 101, and multiple external warehouses (such as Figure 1 103a and 103b in the figure, collectively referred to as 103) and a plurality of cloud object storage warehouses corresponding to the external warehouses 103 (such as Figure 1104a and 104b in the figure are collectively referred to as 104).

[0091] The local warehouse 101 communicates with multiple foreign warehouses 103 via the Internet.

[0092] The local warehouse 101 can be used to store small files. In the embodiment of the present application, small files are called code files. The cloud object storage warehouse 102 corresponding to the local warehouse 101 can be used to store large files. In the embodiment of the present application, large files can be called data files. The storage pressure of the local warehouse 101 is alleviated, and it is not affected by the rapid expansion of the submission of the storage file and the occupation of more disk space caused by the increasing number of submissions corresponding to the modification of large files.

[0093] The cloud object storage warehouse 104 corresponding to the multiple foreign warehouses 103 can be used to store data files, and the data files can be sent by the cloud object storage warehouse 102 corresponding to the local warehouse 101 to the multiple foreign warehouses 103, so that the multiple foreign warehouses 103 can store the data files in the cloud object storage warehouse 104 corresponding to the multiple foreign warehouses 103 through object storage. The intranet clients of the multiple foreign warehouses 103 can download data files from the cloud object storage warehouse 104 corresponding to the foreign warehouse 103 in their location, without downloading data files from the cloud object storage warehouse 102 corresponding to the local warehouse 101 through the Internet, saving the download time of users.

[0094] In a possible implementation, the local warehouse 101 and the multiple foreign warehouses 103 may include a storage scheduling server (such as Figure 1 105a, 105b and 105c are collectively referred to as 105), wherein the storage scheduling server 105a is set in the local warehouse 101, and the storage scheduling servers 105b and 105c are respectively set in multiple foreign warehouses 103, and the storage scheduling server 105 is used to process the client's data storage request and data acquisition request. For example, the storage scheduling server 105a of the local warehouse 101 can be used to schedule the storage of files to be stored in the local warehouse 101 or the cloud object storage warehouse 102 corresponding to the local warehouse. The storage scheduling server 105b in the foreign warehouse 103a can be used to process the data acquisition request of the client (for example, the intranet client), so that the client can download data in the cloud object storage warehouse 104a corresponding to the foreign warehouse 103a or download data in the local warehouse 101.

[0095] In the embodiment of the present application, "warehouse" refers to a small data center formed by dividing the data center 10, the local warehouse 101 is the main data center of the data center 10, and the multiple foreign warehouses 103 can be other small data centers located in other locations of the data center 10. Exemplarily, taking the local warehouse 101 as an example, the local warehouse 101 can store devices, including servers and network devices, etc., wherein the server is the core component of the local warehouse 101, which is used to run applications and process data, and the network devices include routers, switches, firewalls and load balancers, etc., which are used to manage and optimize data flows and ensure security and efficient network connections.

[0096] The cloud object storage warehouse may be deployed in the cloud. For example, the cloud object storage warehouse may include a storage server and a network device, wherein the storage server may be a physical server for storing data objects. The storage server is usually equipped with a large number of hard disk drives (HDD) or solid state drives (SSD) to provide the required storage capacity and performance. The network device may include switches, routers, and network interface cards (NIC), which together build an efficient data transmission network.

[0097] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0098] Figure 2 The present invention provides a flow chart of a data storage method for a data center, which can be executed by a data storage device of the data center, which can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The storage scheduling server 105a in the local warehouse 101 shown, hereinafter referred to as the server, is Figure 2 As shown, the method may include the following steps.

[0099] S201. Receive a storage request from a first client, where the storage request is used to request to store multiple files.

[0100] The server may receive a storage request sent by the first client, and the storage request may be sent by the user through the first client via the Internet. That is, for the local warehouse, the first client is an external network client.

[0101] For example, the first client may be a user's personal computer (PC), user equipment, etc. The user may request the server to store multiple files through the first client, so as to store the multiple files in the data center. It can be understood that the data center is a remote storage center relative to the first client, and the corresponding files can be downloaded or uploaded through the data center.

[0102] S202: Classify multiple files to obtain data files and code files.

[0103] After the server obtains the storage request, it can classify the multiple files to obtain data files and code files. Exemplarily, the number of data files and code files can be multiple.

[0104] Exemplarily, the data file may be a large file, for example, the storage capacity occupied may be greater than or equal to a preset threshold. The code file may be a small file, for example, the storage capacity occupied may be less than a preset threshold.

[0105] S203: Store the data file in the cloud object storage warehouse corresponding to the local warehouse, and store the pointer files of the code file and the data file in the local warehouse, where the pointer files are used to indicate the storage address of the data file.

[0106] The server can store the data files in the cloud object storage warehouse corresponding to the local warehouse, and store the pointer files of the code files and the data files in the local warehouse, where the pointer files are used to indicate the storage address of the data files.

[0107] In a possible implementation, if there is a data file among the multiple files, and the first client uploads the multiple files by ordinary upload, the server may send an instruction message to the first client, the instruction message is used to instruct to upload the data file by object storage, so that the data file is stored in the cloud object storage warehouse corresponding to the local warehouse. The ordinary upload method may be a method in which the first client uploads the code file to the local warehouse.

[0108] For example, Figure 3 A schematic diagram of the storage code files and data files of this application example, such as Figure 3As shown, for a data file in a plurality of files, the first client can use an object storage service to store the data file in an object storage warehouse corresponding to a local warehouse, wherein the object storage service refers to storing the data in the data file as an object, each object includes the data itself, corresponding metadata and a unique identifier, wherein the metadata is used to represent the properties of the object, such as creation time, size, file type, and a more detailed description of the object storage service is not introduced in this application. These objects are placed in a flat data environment called a storage pool, which enables users to quickly and easily obtain the required data, while the flat data environment enables users to quickly expand when large storage requirements are required.

[0109] When a user downloads data, the object storage service can use the unique identifier and metadata to find the required object. For example, the metadata of the required object can be queried through the representational state transfer application program interface (RESTful API) and the object can be obtained based on the metadata, where the RESTful API uses the standard hypertext transfer protocol (HTTP) / secure hypertext transfer protocol (HTTPS). At the same time, the object storage service supports parallel retrieval, which can speed up data access.

[0110] Exemplarily, the pointer file may include a unique identifier of the object, and the cloud object storage warehouse corresponding to the local warehouse may search for corresponding metadata based on the unique identifier, and search for the storage location of the object based on the metadata, thereby achieving data acquisition.

[0111] In this embodiment, the server can receive a storage request from the first client, where the storage request is used to request the storage of multiple files. The multiple files can then be classified to obtain data files and code files, and the data files can be stored in a local object storage warehouse, as well as the pointer files of the code files and data files can be stored in the local warehouse. This allows large files (i.e., data files) to be stored in the cloud, alleviating the storage pressure on the local warehouse.

[0112] Next, it is explained how the server determines the data files and code files.

[0113] Figure 4A flowchart of another data storage method for a data center provided in an embodiment of the present application, the method can be executed by a data storage device of a data center, and the data storage device of the data center can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The storage scheduling server 105a in the local warehouse 101 shown, hereinafter referred to as the server, is Figure 4 As shown, the method may include the following steps.

[0114] S401: Receive a storage request from a first client, where the storage request is used to request storage of multiple files.

[0115] S402: Determine the storage capacity occupied by multiple files.

[0116] After the server obtains the storage request, it can determine the occupied storage capacity of the multiple files in the storage request, that is, determine the file sizes of the multiple files.

[0117] S403: Determine, among the multiple files, a file whose storage capacity is greater than or equal to a preset threshold as a data file.

[0118] The server may determine, among multiple files, files that occupy a storage capacity greater than or equal to a preset threshold as data files. Exemplarily, the preset threshold may be 100 MB (megabytes). Exemplarily, the data files may be large files such as image files, video files, binary files, etc.

[0119] S404: Determine, among the multiple files, a file whose storage capacity is smaller than a preset threshold as a code file.

[0120] The server may determine, among the multiple files, a file that occupies a storage capacity less than a preset threshold as a code file. Exemplarily, the code file may be a small file such as a code script.

[0121] S405: Store the data file in the cloud object storage warehouse corresponding to the local warehouse, and store the pointer files of the code file and the data file in the local warehouse.

[0122] In this embodiment, multiple files to be stored can be classified according to file size, so that different types of files can be stored in different warehouses, alleviating the storage pressure of the local warehouse.

[0123] Furthermore, after storing the data files in the cloud object storage warehouse corresponding to the local warehouse, the data files can be synchronously stored in the cloud object storage warehouse corresponding to other foreign warehouses, so that the intranet clients corresponding to other warehouses can pull data files from other warehouses, which can reduce the transmission time of pulling data files.

[0124] Specifically, Figure 5 A flowchart of another data storage method of a data center provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method may include the following steps.

[0125] S501. The storage scheduling server of the local warehouse may determine a tree structure between the local warehouse and multiple foreign warehouses, and configure the tree structure in a cloud object storage warehouse corresponding to the local warehouse.

[0126] Among them, the local warehouse is the main node of the tree structure, and multiple foreign warehouses are child nodes of the tree structure, and the child nodes include multi-level child nodes.

[0127] For example, Figure 6 This is a schematic diagram of a tree structure exemplified in this application, where the local warehouse is the main node (i.e., the root node), and multiple foreign warehouses are child nodes, where the child nodes are divided into two levels of child nodes. Taking child node 11 as an example, the main node 1 can synchronize data files to child nodes 11, 12, and 13, and child node 11 can synchronize data files to child nodes 21 and 22. S502, the cloud object storage warehouse corresponding to the local warehouse synchronously sends data files to the storage scheduling server of the foreign warehouse corresponding to the next level child node of the main node in the multi-level child node based on the tree structure.

[0128] exist Figure 5 In the multi-level sub-nodes, the foreign warehouse corresponding to the next-level sub-node of the main node is abbreviated as the foreign warehouse.

[0129] The storage scheduling server of the local warehouse can send data files to the storage scheduling server of the foreign warehouse corresponding to the next-level child node of the main node in the multi-level child node based on the tree structure. Figure 6 For example, the next-level child nodes of the main node in the multi-level child nodes are child nodes 11, child nodes 12 and child nodes 13 respectively. The storage scheduling server of the local warehouse can send data files to the storage scheduling servers of the foreign warehouses corresponding to child nodes 11, child nodes 12 and child nodes 13 based on the tree structure.

[0130] Exemplarily, the storage scheduling server of the local warehouse may send data files to the storage scheduling servers of the foreign warehouses corresponding to the sub-nodes 11, 12, and 13 in a distribution manner of object storage.

[0131] S503. The storage scheduling server of the foreign warehouse corresponding to the next-level child node of the master node in the multi-level child nodes stores the data file in the corresponding cloud object storage warehouse.

[0132] It can be understood that the foreign warehouse corresponding to the next-level child node of the master node in the multi-level child node can synchronously send the data file to the foreign warehouse corresponding to the next-level child node of the next-level child node, so as to Figure 6 For example, the next level child nodes of child node 11, child node 12 and child node 13 are child node 21, child node 22, child node 23, child node 24, child node 25 and child node 26 respectively.

[0133] After receiving the data file, the foreign warehouse corresponding to the next-level child node of the main node in the multi-level child nodes can store the data file in the corresponding cloud object storage warehouse.

[0134] Specifically, the data files can be stored in the corresponding cloud object storage warehouse using object storage through a storage scheduling server in an external warehouse corresponding to a next-level child node of a main node in a multi-level child node.

[0135] In one possible implementation, when an external warehouse corresponding to a next-level child node of a main node in a multi-level child node stores a data file in a corresponding cloud object storage warehouse, it can generate a pointer file corresponding to the data file and store it in the external warehouse to facilitate subsequent acquisition of the data file based on the pointer file.

[0136] In this embodiment, data files can be synchronously stored in other foreign warehouses in a tree structure to improve synchronization efficiency, and the intranet clients corresponding to other foreign warehouses can pull data files from other foreign warehouses, which can reduce the transmission time of pulling data files.

[0137] The following describes how to obtain code files and data files when a client initiates a request to obtain files from an external warehouse.

[0138] Figure 7 A flowchart of another data storage method for a data center provided in an embodiment of the present application is provided. The method can be executed by a data storage device of a data center, which can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or an electronic device. The electronic device can be, for example, Figure 1 The storage scheduling server 105a in the local warehouse 101 shown, hereinafter referred to as the server, is Figure 7 As shown, the method may include the following steps.

[0139] S701. Receive an acquisition request corresponding to a second client, where the second client is an intranet client of a target foreign warehouse among multiple foreign warehouses.

[0140] The server may receive a corresponding acquisition request from a second client, where the second client is an intranet client of a target foreign warehouse among the multiple foreign warehouses. It should be understood that the second client being an intranet client of the target foreign warehouse means that the second client and the target foreign warehouse may be in the same local area network.

[0141] S702: If the acquisition request is for requesting a code file, send a target file corresponding to the acquisition request to the second client.

[0142] If the acquisition request is used to request a code file, the server can obtain the target file in the local repository according to the acquisition request, and send the target file to the second client.

[0143] If the acquisition request is used to request a code file, the acquisition request may be sent by the second client.

[0144] S703: If the acquisition request is for requesting a data file, determine the target domain name address and send the target domain name address to the target foreign warehouse.

[0145] The target domain name address is used to obtain a pointer file of the target file, and the pointer file of the target file is used to obtain the target file.

[0146] If the acquisition request is used to request a data file, the acquisition request may be sent by the storage scheduling server of the target foreign warehouse. The acquisition request may be sent by the second client to the storage scheduling server of the target foreign warehouse, and the storage scheduling server of the target foreign warehouse may forward the acquisition request to the storage scheduling server in the local warehouse.

[0147] If the acquisition request is used to request a data file, the server can determine the target domain name address and send the target domain name address to the target foreign warehouse, so that the target foreign warehouse can resolve the target domain name address, obtain the resolved Internet protocol (IP) address, and obtain the pointer file of the target file based on the IP address.

[0148] In this embodiment, the intranet client of other external warehouses can obtain data files and code files through the local warehouse, where the code files can be downloaded directly from the local warehouse, and the data files need to return the target domain name address to the external warehouse, so that the client can obtain the data files from the corresponding cloud object storage warehouse based on the target domain name address.

[0149] In a possible implementation, the server can determine whether the target cloud object storage warehouse has the target file corresponding to the acquisition request, and the target cloud object storage warehouse is the cloud object storage warehouse corresponding to the target foreign warehouse. If so, the target domain name address can be determined to be the first domain name address, and the first domain name address is the domain name address of the target cloud object storage warehouse. If not, the target domain name address can be determined to be the second domain name address, and the second domain name address is the domain name address of the cloud object storage warehouse corresponding to the local warehouse.

[0150] That is to say, if the target cloud object storage warehouse has the target file corresponding to the acquisition request, the second client can be instructed to obtain the target file in the target cloud object storage warehouse to reduce the time for downloading the target file. Specifically, the target domain name address can be determined to be the first domain name address, so that the resolution server of the target foreign warehouse can resolve the first domain name address and obtain the IP address of the local LAN of the target foreign warehouse, so that the second client can obtain the pointer file of the target file from the target foreign warehouse based on the IP address, and obtain the target file from the target cloud object storage warehouse according to the pointer file.

[0151] If the target cloud object storage warehouse does not have the target file corresponding to the acquisition request, the second client can be instructed to obtain the target file from the cloud object storage warehouse corresponding to the local warehouse to meet the user's acquisition needs. Specifically, the target domain name address can be determined to be the second domain name address, so that the resolution server of the target foreign warehouse can resolve the second domain name address and obtain the IP address of the local warehouse, so that the second client can obtain the pointer file of the target file from the local warehouse based on the IP address, and obtain the target file from the cloud object storage warehouse corresponding to the local warehouse according to the pointer file.

[0152] In one possible implementation, the server can determine whether the target file exists in the target cloud object storage warehouse by:

[0153] The server may determine whether synchronization and storage of the target file to the target cloud object storage warehouse is completed. If yes, it is determined that the target file exists in the target cloud object storage warehouse, and if no, it is determined that the target file does not exist in the target cloud object storage warehouse.

[0154] It can be understood that if the target file is synchronously stored in the target cloud object storage warehouse, the server will receive a corresponding feedback message, which is used to indicate that the target file has been synchronously stored. Exemplarily, the feedback message can be sent by the storage scheduling server of the target off-site warehouse.

[0155] In this embodiment, since it takes a long time to synchronize and store the target file, it is possible to determine whether the target file exists in the target cloud object storage warehouse by determining whether the synchronization of the target file is complete, thereby avoiding file download errors caused by the target file not existing in the target cloud object storage warehouse due to failure to synchronize the target file.

[0156] Next, through Figure 8 and Fig. 9 Another data storage method for a data center provided in the present application is described.

[0157] Figure 8 A flowchart of another data storage method for a data center provided in an embodiment of the present application is shown below. Fig. 9 The schematic diagram of the data storage method and data acquisition method of the data center of this application example is as follows: Figure 8 As shown, the method may include the following steps.

[0158] S801. A storage scheduling server of a local warehouse receives a storage request from a first client, where the storage request includes multiple files to be stored.

[0159] S802. The storage scheduling server of the local warehouse determines the occupied storage capacity of multiple files.

[0160] S803: The storage scheduling server of the local warehouse determines, among the multiple files, a file whose storage capacity is greater than or equal to a preset threshold as a data file.

[0161] S804: The storage scheduling server of the local warehouse determines, among the multiple files, a file whose storage capacity is less than a preset threshold as a code file.

[0162] S805. The storage scheduling server of the local warehouse stores the pointer files of the code files and the data files in the local warehouse.

[0163] S806. The storage scheduling server of the local warehouse sends the data file to the cloud object storage warehouse corresponding to the local warehouse.

[0164] After storing the data files in the cloud object storage warehouse corresponding to the local warehouse, the storage scheduling server of the local warehouse determines the tree structure between the local warehouse and multiple foreign warehouses.

[0165] S807. The cloud object storage warehouse corresponding to the local warehouse synchronously sends data files to the foreign warehouse corresponding to the next-level child node of the main node in the multi-level child nodes based on the tree structure.

[0166] It is understandable that the tree structure can be configured in advance in the cloud object storage warehouse corresponding to the local warehouse.

[0167] S808. The external warehouse corresponding to the next-level child node of the master node in the multi-level child nodes stores the data file in the corresponding cloud object storage warehouse.

[0168] S809: The storage scheduling server of the local warehouse receives the acquisition request corresponding to the second client.

[0169] S810: If the acquisition request is for requesting a code file, the storage scheduling server of the local warehouse sends a target file corresponding to the acquisition request to the second client.

[0170] S811. If the acquisition request is for requesting a data file, the storage scheduling server of the local warehouse determines the target domain name address.

[0171] S812: The storage scheduling server of the local warehouse sends the target domain name address to the storage scheduling server of the target foreign warehouse, so that the resolution server of the target foreign warehouse resolves the target domain name address to obtain the target IP address.

[0172] If the target domain name address is the first domain name address, the target IP address is the IP address of the local LAN of the target foreign warehouse, and if the target domain name address is the second domain name address, the target IP address is the IP address of the local warehouse. The resolution server of the target foreign warehouse sends the target IP address to the second client.

[0173] S813. The second client obtains the target IP address, and obtains the target file from the cloud object storage warehouse according to the target IP address.

[0174] For the specific implementation and beneficial effects of the above steps, please refer to the above embodiments, which will not be described again here.

[0175] Fig.10 This is a schematic diagram of a data storage device in a data center provided in an embodiment of the present application. Fig.10 As shown, the device 100 includes: an acquisition module 1001 , a processing module 1002 and a storage module 1003 .

[0176] The receiving module 1001 is used to receive a storage request from a first client, where the storage request is used to request to store multiple files.

[0177] The processing module 1002 is used to classify multiple files to obtain data files and code files.

[0178] The storage module 1003 is used to store the data file in the cloud object storage warehouse corresponding to the local warehouse, and store the pointer file of the code file and the data file in the local warehouse, the pointer file is used to indicate the storage address of the data file.

[0179] Optionally, in the above apparatus 100, the processing module 1002 is specifically configured to:

[0180] Determine the storage capacity occupied by multiple files.

[0181] Among the multiple files, a file that occupies a storage capacity greater than or equal to a preset threshold is determined as a data file.

[0182] A file among the multiple files that occupies a storage capacity smaller than a preset threshold is determined as a code file.

[0183] In a possible implementation, the data center further includes multiple off-site warehouses and cloud object storage warehouses corresponding to the multiple off-site warehouses, and the local warehouse communicates with the multiple off-site warehouses via the Internet.

[0184] The apparatus 100 further includes a synchronization module 1004, which is used to:

[0185] A tree structure between a local warehouse and multiple foreign warehouses is determined, wherein the local warehouse is a main node of the tree structure, and the multiple foreign warehouses are child nodes of the tree structure, and the child nodes include multiple levels of child nodes.

[0186] Based on the tree structure, the cloud object storage warehouse corresponding to the local warehouse synchronously sends the data files to the foreign warehouse corresponding to the next-level child node of the main node in the multi-level child nodes.

[0187] The data files are stored in the corresponding cloud object storage warehouse through the off-site warehouse corresponding to the next-level child node of the main node in the multi-level child node.

[0188] In a possible implementation, the apparatus 100 further includes a download module 1005, and the download module 1005 is used to:

[0189] A corresponding acquisition request from a second client is received, where the second client is an intranet client of a target foreign warehouse among multiple foreign warehouses.

[0190] If the acquisition request is for requesting a code file, a target file corresponding to the acquisition request is sent to the second client.

[0191] If the acquisition request is used to request a data file, the target domain name address is determined and sent to the target foreign warehouse. The target domain name address is used to obtain a pointer file of the target file, and the pointer file of the target file is used to obtain the target file.

[0192] In a possible implementation, the download module 1005 is specifically used for:

[0193] Determine whether a target file corresponding to the acquisition request exists in a target cloud object storage warehouse, where the target cloud object storage warehouse is a cloud object storage warehouse corresponding to the target off-site warehouse.

[0194] If so, it is determined that the target domain name address is the first domain name address, and the first domain name address is the domain name address of the target cloud object storage warehouse.

[0195] If not, it is determined that the target domain name address is the second domain name address, and the second domain name address is the domain name address of the cloud object storage warehouse corresponding to the local warehouse.

[0196] In a possible implementation, the download module 1005 is specifically used for:

[0197] Determine whether the synchronization of the target file to the target cloud object storage warehouse is complete.

[0198] If so, it is determined that the target file exists in the target cloud object storage warehouse.

[0199] If not, it is determined that the target file does not exist in the target cloud object storage repository.

[0200] The data storage device of the data center provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be described in detail here.

[0201] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the electronic device 110 may include: at least one processor 1101 and a memory 1102 .

[0202] The memory 1102 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0203] The memory 1102 may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0204] The processor 1101 is used to execute the computer-executable instructions stored in the memory 1102 to implement the method described in the above method embodiment. The processor 1101 may be a CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0205] Optionally, the electronic device 110 may further include: a communication interface 1103. In a specific implementation, if the communication interface 1103, the memory 1102 and the processor 1101 are implemented independently, the communication interface 1103, the memory 1102 and the processor 1101 may be interconnected through a bus and communicate with each other. The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0206] Optionally, in a specific implementation, if the communication interface 1103, the memory 1102 and the processor 1101 are integrated on a chip, the communication interface 1103, the memory 1102 and the processor 1101 can communicate through an internal interface.

[0207] The electronic device 110 may be a server, such as a storage scheduling server of a local warehouse or a storage scheduling server of multiple foreign warehouses.

[0208] The electronic device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0209] An embodiment of the present application provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a disk or an optical disk, and other media that can store computer-executable instructions. Specifically, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0210] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0211] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0212] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0213] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0214] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0215] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processing technology (DSP) chip, and an application-specific integrated circuit (ASIC), etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0216] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0217] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0219] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data storage method for a data center, characterized in that: The data center includes a local warehouse and a cloud object storage warehouse corresponding to the local warehouse, and the method includes: Receiving a storage request from a first client, wherein the storage request is used to request storage of multiple files; Classify the multiple files to obtain data files and code files; The data file is stored in a cloud object storage warehouse corresponding to the local warehouse, and the code file and the pointer file of the data file are stored in the local warehouse, wherein the pointer file is used to indicate the storage address of the data file.

2. The method according to claim 1, characterized in that: The multiple files are classified to obtain data files and code files, including: Determining the occupied storage capacity of the plurality of files; Determine, among the multiple files, a file whose storage capacity is greater than or equal to a preset threshold as the data file; A file among the multiple files that occupies a storage capacity smaller than the preset threshold is determined as the code file.

3. The method according to claim 1 or 2, characterized in that: The data center further includes a plurality of off-site warehouses and cloud object storage warehouses corresponding to the plurality of off-site warehouses, and the local warehouse communicates with the plurality of off-site warehouses via the Internet; The method further comprises: Determine a tree structure between the local warehouse and the multiple foreign warehouses, wherein the local warehouse is a main node of the tree structure, and the multiple foreign warehouses are child nodes of the tree structure, and the child nodes include multiple levels of child nodes; The data file is synchronously sent to the foreign warehouse corresponding to the next-level child node of the master node in the multi-level child nodes through the cloud object storage warehouse corresponding to the local warehouse based on the tree structure; The data file is stored in the corresponding cloud object storage warehouse through the external warehouse corresponding to the next level child node of the main node in the multi-level child nodes.

4. The method according to claim 3, characterized in that The method further comprises: receiving a corresponding acquisition request from a second client, where the second client is an intranet client of a target foreign warehouse among the multiple foreign warehouses; If the acquisition request is for requesting a code file, sending the target file corresponding to the acquisition request to the second client; If the acquisition request is used to request a data file, the target domain name address is determined and sent to the target foreign warehouse. The target domain name address is used to obtain a pointer file of the target file, and the pointer file of the target file is used to obtain the target file.

5. The method according to claim 4, characterized in that Determine the target domain name address, including: Determine whether the target file corresponding to the acquisition request exists in the target cloud object storage warehouse, the target cloud object storage warehouse being the cloud object storage warehouse corresponding to the target foreign warehouse; If so, determining that the target domain name address is a first domain name address, the first domain name address is the domain name address of the target cloud object storage warehouse; If not, it is determined that the target domain name address is a second domain name address, and the second domain name address is the domain name address of the cloud object storage warehouse corresponding to the local warehouse.

6. The method according to claim 4 or 5, characterized in that: Determining whether the target cloud object storage warehouse has the target file corresponding to the acquisition request includes: Determining whether synchronization and storage of the target file to the target cloud object storage warehouse is completed; If so, determining that the target file exists in the target cloud object storage repository; If not, it is determined that the target file does not exist in the target cloud object storage repository.

7. A data storage device for a data center, characterized in that: The data center includes a local warehouse and a cloud object storage warehouse corresponding to the local warehouse, and the device includes: A receiving module, configured to receive a storage request from a first client, wherein the storage request is used to request storage of multiple files; A processing module, used for classifying the multiple files to obtain data files and code files; A storage module is used to store the data file in a cloud object storage warehouse corresponding to the local warehouse, and to store the code file and the pointer file of the data file in the local warehouse, wherein the pointer file is used to indicate the storage address of the data file.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images